Hey, we need an FTP server. Yes, it has to be an FTP server. We’re going to tell our clients to upload files there. Oh, and we don’t want to bother creating userids for each individual clients—too much work, so it’s important that they can upload files but can’t see anybody else’s files. Except for us: we need to be able to see all the uploaded files.

FTP, although long since superseded by better & more efficient file transfer protocols (e.g. scp, HTTP, bittorrent), has managed to survive to this day, occasionally rearing its ugly head, reminding us that dinosaurs still walk the earth.

Here are the steps to go through to create a secure anonymous FTP server, one where the anonymous clients can upload files but cannot read them.

Create an Amazon EC2 Instance

Amazon AWS is an excellent service for hosting virtual machines on the Internet. Create an account and perform the following steps (caveat lector: Amazon may change the menus/procedures at its discretion):

• Log into Amazon AWS (you have created an account, haven’t you?)
• Click on EC2 Dashboard
• Click Launch Instance
• Select Classic Wizard
• Select Ubuntu Server 12.04.1 LTS 64-bit
• Select EC2; if you have an Availability Zone preference, select it here.
• The Advanced Instance Options have a reasonable set of defaults
• Similarly with the Storage Device Configuration
• Add Tags
  • Name: anon FTP
• Select Create a new Key Pair,
  • name it anonftp
  • click Create & Download
  • save to ~/.ssh/anonftp.pem
• Select Create a new Security Group
  • group name anon FTP
  • group description allow all TCP
  • Create a new rule: All TCP
  • Click Add Rule
• Click Launch

• Click Close

• Click on Elastic IPs

• Click Allocate New Address (for example, 54.243.47.142, which we will use in the remainder of this document, but remember to substitute your allocated elastic IP address)
• Click Associate Address
  • Instance: anon FTP

Configure Anon FTP

chmod 600 ~/.ssh/anonftp.pem
ssh -i ~/.ssh/anonftp.pem ubuntu@54.243.47.142
sudo apt-get install vsftpd
sudo vim /etc/vsftpd.conf

We made the following changes to our /etc/vsftpd.conf file:

local_enable=YES
write_enable=YES
anon_upload_enable=YES
chown_uploads=YES
chown_username=ftpmaster
chroot_local_user=YES

Now we need to create our ftpmaster user, who will be able to log in & see all files. We are going to assign him the password themasterseesall:

sudo restart vsftpd
sudo useradd -G ftp -d /srv/ftp ftpmaster
sudo passwd ftpmaster
sudo mkdir /srv/ftp/pub
sudo chown ftpmaster:ftp /srv/ftp/pub
sudo chmod 733 /srv/ftp/pub
sudo tee /srv/ftp/readme.txt <<-EOF
        Please upload all movies into the /pub directory.

        You may upload files into the pub directory, but you will not be
        able to read files in the pub directory, not even the ones
        you've uploaded.

        If you can't upload files into the /pub directory, it is possible
        that there is already a file of the same name already there; try
        uploading your file using a different name.
EOF

Test

From your workstation (not the Amazon EC2 instance), connect via anonymous FTP & upload a file. Also, try to get a directory listing:

ftp ftp@54.243.47.142
Connected to 54.243.47.142.
220 (vsFTPd 2.3.5)
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> put /etc/hosts /pub/hosts.txt
local: /etc/hosts remote: /pub/hosts.txt
229 Entering Extended Passive Mode (|||64563|).
150 Ok to send data.
100% |*********************************************************************************************|   236        4.50 MiB/s    00:00 ETA
226 Transfer complete.
236 bytes sent in 00:00 (1.44 KiB/s)
ftp> ls /pub/
229 Entering Extended Passive Mode (|||29280|).
150 Here comes the directory listing.
226 Transfer done (but failed to open directory).
ftp> quit
221 Goodbye.

That was a successful FTP session:

• we were able to connect & upload the file
• we were unable to browse the contents of the upload directory

Now let’s make sure that the ftpmaster can log in & retrieve the uploaded files:

• Browse ftp://ftpmaster:themasterseesall@54.243.47.142/pub
• Click on the hosts.txt file to download

Configure DNS

• Create a DNS A record for anonftp.yourcompany.com that points to 54.243.47.142

Security Concerns

Anonymous FTP can be a security concern (this author ran an anonymous FTP server in 2001 only to discover that a gentleman from Germany was using its diskspace & bandwidth to illegally distribute movies. Even worse, his taste in movies was universally mediocre). But this should not be a concern: given that anonymous FTP users cannot see or download the material they have uploaded, our German hacker would be thwarted in his attempt to use this FTP server as a distribution mechanism. Also, bittorrent has supplanted using pilfered anonymous FTP servers in the modern day.

There are also [somewhat lame] denial-of-service attacks: someone could, for example, fill the up the disk space, preventing others from uploading.

There is no encryption on the FTP uploads. If the content is sensitive, this may not be the appropriate solution.

Can the server be hacked? Can someone break in through one of the services and own the machine? I suspect the likelihood is low: there are only 2 services running: FTP & ssh.

• The only user who can ssh in is the ubuntu user (for the sshd configuration requires ssh keys to log in, and ubuntu is the only user that has keys).
• The FTP service is using an FTP server that has been built for security.

Helps

Adding an “external” dependency

The heroku_san gem used to depend on the Heroku CLI gem, ‘heroku’ which has been sunsetted in favor of the Heroku Toolbelt. What’s a reliable way of including this dependency in the gem library?

Interestings

Smallest Federated Wiki

Ward Cunningham, inventor of the Wiki, has been working on a sequel called “Smallest Federated Wiki”. It’s packed with novel ideas about collaboration, markup, mobile emphasis, etc.

Best resources so far:
Presentation at RealTimeConf

videos:

http://wardcunningham.github.com/

New Rails and Ruby patches

patch 327 is avail. for Ruby: http://www.ruby-lang.org/en/news/2012/11/09/ruby19-hashdos-cve-2012-5371/

rails 3.2.9 is out this morning: https://github.com/rails/rails/tree/3-2-stable

I’m a big fan of David Dollar’s foreman for starting up everything you need to run an app. These days were seeing a number of projects that rely on multiple ruby apps with seperate gemfiles to have a usuable development environment.

On one project recently we had three external services which we did not want to depend on for development. We wrote a tiny rails app for each, which had json endpoints, models for fake data, and used Rails Admin almost exclusively for the UI. This was great, but starting each one by hand was a chore, and foreman didn’t want to respect the individual gemfiles.

The first thing that makes this dificult is bundler, which attempts to be helpful and make sure your gemset is still in effect when you shell out. How it does this is an ugly or beautiful hack, depending on your point of view.

    1.9.3p194 :001 > ENV['RUBYOPT']
    => nil
    1.9.3p194 :002 > require 'bundler'
    => true
    1.9.3p194 :003 > Bundler.require
    => [...]
    1.9.3p194 :004 > ENV['RUBYOPT']
    => "-I/Users/mkocher/.rvm/gems/ruby-1.9.3-p194@soloist/gems/bundler-1.2.0.rc.2/lib -rbundler/setup"

As you can see, Bundler has set RUBYOPT to load itself every time you shell out. We can prevent this from happening by using Bundler.with_clean_env.

    1.9.3p194 :005 > Bundler.with_clean_env { ENV['RUBYOPT'] }
    => nil

Better! But we’re not out of the woods yet.

RVM has recently started including rubygems-bundler in the global gemset. I with they hadn’t, but I also don’t want to tell people to uninstall it every time they get a new workstation.

To prevent rubygems-bundler from trying to keep you in your gemset, you’ll want to add a NOEXEC=skip to your environment.

With ruby, the cleanest way to do this is to include the env hash in your call to system:

  system({'NOEXEC'=>'skip'}, "rake -T")

Putting this all together, we made a script called run_app which looks like

    #!/usr/bin/env ruby
    require 'bundler'

    cmd = %Q{bash -lc "cd ../#{ARGV[0]} && source #{ENV['rvm_path']}/scripts/rvm && exec rackup config.ru -p #{ARGV[1]}"}

    Bundler.with_clean_env do
      system({'NOEXEC'=>'skip'}, cmd)
    end

We call this from our main project, and can pass it a directory and the port we want it to run on. foreman start now gives us an entire development environment nicely logging to one terminal.

Splunk is capable of authenticating users against LDAP, including Apple’s Open Directory.

To configure Splunk to authenticate against Apple’s Open Directory, start by logging into Splunk and creating a new LDAP strategy by navigating to the following:
Manager → Access controls → Authentication method

• Check LDAP
• Click Configure Splunk to use LDAP and map groups
• Click New
• Enter the below settings:

  LDAP strategy name: opendirectory

  Host: opendirectory.sf.pivotallabs.com
  Port: 389
  SSL: unchecked
  Bind DN: uid=diradmin,cn=users,dc=opendirectory,dc=sf,dc=pivotallabs,dc=com
  Bind DN Password: Open Directory diradmin password
  Confirm Password: Open Directory diradmin password

  User base DN: cn=users,dc=opendirectory,dc=sf,dc=pivotallabs,dc=com
  User base filter: blank
  User name attribute: uid
  Real name attribute: cn
  Group mapping attribute: uid

  Group base DN: cn=groups,dc=opendirectory,dc=sf,dc=pivotallabs,dc=com
  Static group search filter: blank
  Group name attribute: cn
  Static member attribute: memberuid
  Nested groups: unchecked

  Dynamic member attribute: blank
  Dynamic group search filter: blank

• Click Save
• Click Map groups
• Select the group containing the people who should have access (in our case, “admin”)
• Click add all >>
• Click Save
• Test by trying to log in as an LDAP / OD user from the admin group

Done!