Technology Musings

July 14, 2018

Platforms / Deploying a Rails Application to Elastic Beanstalk


I'm not a big AWS fan because it is a lot of work.  However, sometimes a client requires it.  In any case, if you are wanting to deploy to AWS Elastic Beanstalk using a Rails app, here are your step-by-step instructions!

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Helvetica Neue'; color: #454545} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Helvetica Neue'; color: #454545; min-height: 14.0px} li.li1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Helvetica Neue'; color: #454545} span.s1 {color: #e4af0a} ol.ol1 {list-style-type: decimal}

Preparing a Rails app for EB deployment:


  1. Install the elastic beanstalk tools:
    From Python: sudo pip install awscli awsebcli --upgrade
    From homebrew: brew install aws-elasticbeanstalk
  2. AWS Config
    Set ~/.aws/config as follows:

    region = us-east-1

    [profile eb-cli]
    aws_access_key_id = YOUR_ACCESS_KEY_ID
    aws_secret_access_key = YOUR_SECRET

    The name of the profile "eb-cli" will need to match future steps.
  3. Create an AWS EB application in the EB GUI:
    1. Create the application and give it a name
    2. Create an environment (select "web server environment") and name it whatever you want
    3. Select "Ruby" for the preconfigured platform
    4. Leave the application as "sample application"
    5. Hit "Create Environment" and wait for it to finish (this takes forever)
  4. Enable load balancing so that you have more control (optional, but required for HTTPS I think):
    1. Go to "configuration" then "Capacity"
    2. Change "Single Instance" to "Load Balanced"
    3. Reduce "max" to "1" to prevent additional instances from launching
    4. Hit "apply" and "confirm" and then wait (this takes forever)
  5. Change the instance type to "t2.small" at the lowest.  If it is smaller than this, you will get strange errors when deploying because it is running out of RAM to do the deployment.
    1. "Configuration" -> "Instances" -> "Modify" -> "Instance Type".  Set to t2.small.
  6. Add a data tier
    1. "Configuration" -> "Database" -> "Modify"
    2. Options:
      Engine: postgres
      Engine version: 9.6.8
      Username/password: set them and write them down, but you don't really need them
  7. Configure Environment Variables
    1. "Configuration" -> "Software" -> "Modify"
    2. BUNDLE_WITHOUT: test:development
    3. RACK_ENV: production
    7. SECRET_KEY_BASE: use "rake secret" to generate one (note that leaving this out can lead to some weird errors that seem senseless)
    8. Any other configuration (check your .env file if you have one)
  8. Modify your app to use your data tier.  In config/database.yml, make sure that the adapter is postgresql for "default", then set production to be like the following:

      <<: *default
      database: <%= ENV['RDS_DB_NAME'] %>
      username: <%= ENV['RDS_USERNAME'] %>
      password: <%= ENV['RDS_PASSWORD'] %>
      host: <%= ENV['RDS_HOSTNAME'] %>
      port: <%= ENV['RDS_PORT'] %>

  9. In your app, Create a .elasticbeanstalk/config.yml file in your project's home directory
    1. File contents:

        application_name: your-app-name
        default_region: us-east-1
        profile: eb-cli
          environment: your-staging-environment

    2. The region should probably match the region of the application (though I am not certain), the profile should match the one in ~/.aws/config, and the application_name should match the actual name of the application in elastic beanstalk.
    3. "branch-defaults" allows you to set some parameters per-branch (based on the git branch).  Here, we are saying that when using the master git branch, the environment we should push to is "your-staging-environment".
  10. Create a file in your app .ebextensions/010_packages.config (loads packages we will probably need - may need others based on requirements):

        git: []
        gcc: []
        libxml2-devel: []
        libxslt-devel: []
        patch: []
        sqlite-devel: []
        libffi-devel: []
        postgresql96-devel: []

  11. Make sure that everything in your repository is committed and ready to go.  Then, run

    eb deploy

    and everything should work.
  12. Add a CNAME for your environment in your DNS manager, whatever that is.  It should CNAME to the name listed in your environment (i.e., (optional - you can also just leave it with the generated name, but then you won't be able to add SSL).
  13. Turn on SSL (optional)
    1. Go to AWS Certificate Manager
    2. Click "Request a Certificate" -> "Request a Public Certificate" 
    3. Enter in your domain name (the CNAME you created)
    4. Choose "DNS Validation"
    5. When it says "Pending Validation", open up the "domain" area and create the requested CNAME in your DNS in order to validate your domain.  When complete, click "Continue"
    6. Go to your EB environment page
    7. Go to "Configuration" -> "Load Balancer" -> "Modify"
    8. Click "Add Listener":
      Listener Port: 443
      Listener Protocol: HTTPS
      Instance Port: 80
      Instance Protocol: HTTP
      SSL Certificate: select the certificate previously generated
    9. Note that after adding the listener, there is an "apply" button buried on the webpage.
    10. You can now access the site using SSL on the CNAME you created.


December 22, 2015

Platforms / Rails and Updated Postgres not Performing Migrations


Apparently, there was a change to something in postgres 9.5, or maybe just my install of it.  If you upgrade your Postgres database version and suddenly your rails app stops doing migrations, add the following line to config/database.yml database config to get it working again:


  schema_search_path: public

The error message i was getting was:


ActiveRecord::StatementInvalid: PG::DuplicateTable: ERROR:  relation "schema_migrations" already exists

Basically, sometimes Rails was finding the schema_migrations table, and sometimes it wasn't, but setting the schema_search_path fixed it.

I'm still not sure where exactly the problem came from, but setting the setting fixed it.

June 11, 2014

Platforms / New Library for Reading Pickle Files from Ruby


This is announcing the release of "pickle-interpreter", a new ruby gem for reading Python pickle files in ruby and rails projects.  The project is available at

It is called "pickle-interpreter" because Pickle is actually a mini-stack-based programming language, so this is an interpreter for the Pickle language.

July 31, 2013

Platforms / Uniqueness Methods for PostgreSQL


Just finished packaging together a set of uniqueness functions for PostgreSQL, mostly focused on Cantor pairing.  Here is the introduction from the documentation:


A collection of functions such as cantor pairing which help make unique identifiers for PostgreSQL.

What is a Cantor Pair? Let's say that you have a table which is uniquely identified by two numeric foreign keys. However, let's say that you need a single, unique key to refer to the row externally. Probably a better approach is to create a new primary key, but, if you can't, you can use a Cantor Pairing function to create a single number from the two.

select cantorPair(23, 50);
-- yields 2751
select cantorUnpair(2751);
-- yields {23,50};

Remember, in Postgres, you can even create indexes on these functions.

create table cantortest(a int, b int, c text, primary key(a, b)); create index on cantortest(cantorPair(a, b)); insert into cantortest values (1, 5, 'hello'); insert into cantortest values (3, 18, 'there');

select cantorPair(3, 18); -- =>  (yields 249)

select * from cantortest where cantorPair(a, b) = 249; -- uses index on large enough tables, or if enable_seqscan set to no.

select * from cantortest where cantorUnpair(249)[1] = 3; -- parses 
number out into a foreign key that can be looked up on its own

You can use more than two values with an array and the cantorTuple function:

select cantorTuple(ARRAY[5, 16, 9, 25]);
-- yields 20643596022
select cantorUntuple(20643596022, 4); -- NOTE - the 4 tells how many components
-- yields {5,16,9,25}

Additional functions include:

cantorPairBI - the regular cantorPair uses DECIMALs, which can be slow. If you know the final value will fit in a BIGINT, this version should be a little bit faster.

cantorUnpairBI - same as above but for unpairing

uniquenessSpace(key, spaces, space_index) - this is useful for when you need to UNION tables which each have their own PKs, but you want all of the rows to also have their own PKs. It is a super-simple function (key * spaces + space_index), but it is helpful because people can better see what/why you are doing things. So, for instance, if we had an employees tables and a customers table, but wanted a people view that had a unique PK, we can do:

     SELECT uniquenessSpace(employee_id, 2, 0), employee_name from employees
     SELECT uniquenessSpace(customer_id, 2, 1), customer_name from customers;

To retrieve the original ID, use uniquenessUnspace:

select * from employees where employee_id = uniquenessUnspace(person_id, 2, 0); -- use same arguments as used on uniquenessSpace

These are both horrid names, and I am open to renaming them.

I am currently working on another uniquenessSpace, which doesn't require the number of tables to be known, only the index. However, this function is problematic because it relies on exponentials. The current version doesn't work because I need a prime filter. However, the goal is to overcome the problem in the previous example that if the number of tables change (i.e. we add on an additional supplier table or something), then all of the generated keys have to change (because there are more tables, which affects the second parameter). In an ideal setting, you will be able to do:

     SELECT uniquenessSpace(employee_id, 0), employee_name from employees
     SELECT uniquenessSpace(customer_id, 1), customer_name from customers;

And then add on additional tables as necessary, without ever having to worry about changing keys. This can work by generating a key which is p0^employee_id, and p1^customer_id, where p0 is the first prime (2) and p1 is the second prime (3). As you can see, this gets out of hand pretty quickly, but PostgreSQL is surprisingly accomodative to this. Anyway, this is probably a generally bad idea, but I thought it should be included for completeness. I should probably rename this function as well.

May 09, 2013

Platforms / My Beef with Model Validators in Rails


When I first started using Rails, it was a dream come true.  I estimate that I reduced the amount of code I was writing by 60-75%.  This is probably still true.  However, as I've gotten to know the platform better, there are some philosophical issues that I have with Rails.  Sometimes it is more the Rails community and its conventions rather than rails itself, but nonetheless, it continually manifests itself.

My current beef - validators!  Having written lots and lots of code that interacts with lots and lots of systems, I have to say that you should never, ever, ever put in model validators.  Ever.  Validation is a process issue, not a data one.  In theory it should be a data issue, but it never ever works out that way.

Here's what happens.  Let's say I have a telephone number validator for a model.  It makes sure the numbers are in the format (918) 245-2234.  If not, it gives a nice error message.  However, now let's connect that same model to an external system.  Let's now say that the author of the external system was not as picky about phone number formats.  What happens now?  Well, every record that gets migrated from that system that isn't formatted according to your desires will now get kicked out!  And it might not even show up during testing!  In addition, your pretty error messages mean practically nothing on system->system communication.  So, what you have is the system randomly breaking down.

This causes even more problems in long-term maintenance.  I've had many bugs pop up because someone added a validator that broke several processes.  It worked in all of their tests, but they weren't thinking about all of the ways the system was used.  For instance, we decided to lock down the characters used in our usernames a little more.  The programmer decided to implement it as a validator.  Well, the problem is that several users already had usernames that didn't match the validator.  So what happened?  Those records just stopped saving.  Not just when there was someone who was at the terminal, where they could see the error message, but every job started erroring out because of users who had faulty names.

Because of these issues, I *always* do validation in my controller, and *never* in the model.  It is true that there are probably workarounds for all of these issues, but the point is that they are *not* that exceptional!  They are everywhere!  What would be nice is to have validators which could be run optionally, in specific situations.  So, a controller could declare that it was using :data_entry validations, and then the model would only run those validators.  So, it would look something like this:

In the model:

validates_presence_of :email, :for => [:data_entry]

In the controller:

uses_validators :data_entry

One of the issues I have found with dynamic languages such as ruby is that errors arise when "magic" is sprinkled in unexpected places.  Ruby magic is awesome, but when I do something simple and it won't save, it is sometimes hard to even locate where the error is.  Explicitly declaring which validator set you are using allows programmers to more easily see what they are doing, and only apply magic code where it goes.

Someday I'll write a package like this --- when I get extra time :)

May 07, 2013

Platforms /

Why the tabs-vs-spaces debate matters:

Inconsistent indentation: 2 spaces were used for indentation, but the rest of the document was indented using 1 tab.

Everyone - indent with tabs.  Then you can control the indentation using your editor's settings.  DO NOT indent with spaces.  And DO NOT use any programming language or configuration system that cares about indentation.

October 01, 2012

Platforms / Rails, CoffeeScript, and Java


I *hate* the direction that Rails is moving.  It is becoming a bloated piece of garbage.  I got into Rails precisely because Rails got out of my way most of the time, but was easy to use right where I needed it.  ActiveRecord was a thin shell over SQL.  Therefore, if there was a problem, I could easily diagnose it, because I knew SQL.  It added a few bits of magic, but those were usually pretty understandable.  Likewise with the javascript helpers and the like.  If you needed something done, just pop into Javascript and patch it up.

This contrasted heavily with J2EE.  I used J2EE for a while, but it turns out that if you want to use J2EE you have to commit to it like it's a religion.  If you don't know 20 different platform libraries, you can't even do a hello world in J2EE.  When you finally get it working, you don't know what the heck is even going on.  Some magic is happening somewhere, but it is beyond what is plainly visible.  This, and the fact that Ruby is very metaprogramming-ish, was the reason I chose Rails.

However, starting with Rails 3, the Rails team decided that it was simply going to do everything for you, and everything was going to be done in domain-specific languages.  Thus, instead of the required list of languages/platforms being Ruby/Rails/SQL/Javascript, it has balooned out to Ruby/Rails/SQL/Javascript/Coffeescript/SASS/ECO/Node.js, and probably a few others I'm forgetting.  The asset pipeline is kind of cool, but it's yet another place for things to go wrong.  I think the anti-CSS measure of auto-h()ing code is terrible.  If you wanted to go that way, instead you should use Perl's taint feature, and just give errors if you display tainted values.  The idea that random code should transform my strings is horrible.

Finally, I understand the reason for getting rid of the javascripty stuff from rails core, but some of those were really what made the platform really rock.  I think they have missed the mark - they put the ease-of-use in the wrong places, and instead have made sure that no one that doesn't have a relgious commitment to the software stack will ever find it usable.  It has basically gone the way of J2EE, and I am now looking for a new platform to love.

August 15, 2012

Platforms / Autoreleased with No Pool in Place


Before Apple's ARC, you would often have to manage memory semi-manually.  Before operation queues, you would also have to manage setting up autorelease pools on new threads.  If you didn't, you would get an error such as:

autoreleased with no pool in place - just leaking - break on objc_autoreleaseNoPool() to debug

However, the other day, I started getting these myself!  I was a little shocked - what on earth was I doing that required an autorelease pool?  Well, eventually I figured it out.  It turns out that if you have custom +load() functions on your classes (these get called whenever the classes load - I use them for registering handlers and such), this is another situation in which you have to set up your own autorelease pool if you do any object creation. 

So, in your +load() function (or, depending on what you are doing, in functions that +load calls), just wrap it with

@autoreleasepool {
Your Code Here

And it works!

July 13, 2012

Platforms / iOS Bug - insertNewObjectForEntityName: deleteObject and rollback


I've been chasing down a wierd bug where, on an NSManagedObjectContext, the hasChanges method keeps returning YES even after a rollback.  It turns out, if you do the following sequence, hasChanges will *never* be cleared:

MyClass *c = [NSEntityDescription insertNewObjectForEntityName:@"MyClass" inContext:ctx];
//Do other stuff
[ctx deleteObject:c];
// Do other stuff
[ctx rollback];
[ctx hasChanges]; // Returns YES!

In my own case, I got around it by not deleting the object, and just rolling back if I wanted it gone, but that won't work for everyone.  So, I did:

MyClass *c = [NSEntityDescription insertNewObjectForEntityName:@"MyClass" inContext:ctx];
//Do other stuff
[ctx rollback]; // Don't delete! just roll back
//Do other stuff
[ctx rollback]; // Now it works
[ctx hasChanges]; // Returns NO like it should

May 10, 2012

Platforms / iOS - using storyboard relationships for custom view controller containment


One thing that iOS forgot to do with the release of storyboards and iOS 5 is support for custom "relationship" segues.  In addition to normal segues, UINavigationController and UITabBarController have what are called "relationships", which allow you to define containment information for these controllers.  So, for the navigation controller, you define the root view controller, and for the tab bar controller, you define the list of view controllers.  This is all done in a neat interface in the storyboard editor (which is the replacement for interface builder).

Another cool feature in iOS is "containment", which is basically a way of letting you write view controllers which contain other view controllers.  This was not really allowed in iOS 4, but they've added some basic support for it.  Unfortunately, the way that you might want to use containment with storyboarding is through relationships, but iOS doesn't provide a way to do custom relationships in storyboards - or even their default ones on custom view controllers.

My own task was to create a tab bar controller with the tabs at the *top*.  I had a few ideas on how I might approach this problem.  Subclassing UITabBarController didn't seem like a good idea.  So I tried several others.

One possible way was duck typing.  I thought that perhaps I could pull a fast on on the storyboard editor.  What I would do is define my own class that mimicked UITabBarController without actually inheriting from it.  Then, I could create my relationships with the storyboard editor, and then substitute my own non-tab-bar controller at the last minute.  This failed miserably.  

Firts of all, the storyboard editor doesn't like you to substitute non-subclasses for the class that it wants.  If you try to set it to a non-subclass of UITabBarController, it won't give you an error, but it won't save the change, either.

I tried to work around this by making my class inherit from UITabBarController while I was editing the storyboard, and then change it back for the compile.  This worked momentarily, but in the end, it turns out that the "relationship" segues aren't set via getters and setters.  I imagine there is an internal API that is being used.

Therefore, I next tried overriding initWithCoder and reading the nib values myself.  Unfortunately, I was not successful at guessing the keys for decoding.

Then, it hit me.  What I need to do is to mix subclassing UITabBarController with not mixing UITabBarController.  So here's what I did:

class 1: MyTabBarController

This is just a shell class, but it inherits from UITabBarController.  

class 2: MyTabBarControllerImplementation

This is your real implementation class, and only needs to inherit from UIViewController.  Make it work like you want it to.

What you are going to do is designate your tab bar controllers to be of type MyTabBarController in the storyboard editor.  Then, when viewDidLoad is called on MyTabBarController, the view controller de-registers the view controllers from itself, creates an instance of MyTabBarControllerImplementation, moves the view controllers over there, and then replaces itself with MyTabBarControllerImplementation.

Anyway, here's the code.  It's really rough, but works at the moment.  I'm open to improvements:


#import <UIKit/UIKit.h>
@interface MyTabBarController : UITabBarController


#import "MyTabBarController.h"
#import "MyTabBarControllerImplementation.h"
@implementation JBTabController
- (void)viewDidLoad
    [super viewDidLoad];
    JBTestContainerViewController *vc = [[JBTestContainerViewController alloc] init];
    NSArray *vclist = self.viewControllers;
    [self setViewControllers:[NSArray array] animated:NO];
    vc.viewControllers = vclist;
    NSMutableArray *navlist = [self.navigationController.viewControllers mutableCopy];
    [navlist replaceObjectAtIndex:(navlist.count - 1) withObject:vc];    
    [self.navigationController setViewControllers:navlist animated:NO];



#import <UIKit/UIKit.h>
@interface MyTabBarControllerImplementation : UIViewController<UITabBarDelegate>
@property (nonatomic) int selectedIndex;
@property (strong, nonatomic) NSArray *viewControllers;
@property (strong, nonatomic) UITabBar *bar;



#import "MyTabBarControllerImplementation.h"
@implementation MyTabBarControllerImplementation
@synthesize viewControllers = _viewControllers;
@synthesize selectedIndex = _selectedIndex;
@synthesize bar = _bar;
-(CGRect) subviewFrame {
    CGRect f = self.view.frame;
    return CGRectMake(0, 30, f.size.width, f.size.height - 30);
-(void) reloadBarInfo {
    NSMutableArray *itms = [NSMutableArray arrayWithCapacity:_viewControllers.count];
    for(UIViewController *vc in _viewControllers) {
        UITabBarItem *itm = [[UITabBarItem alloc] initWithTitle:vc.title image:nil tag:0];
        [itms addObject:itm];
        [vc didMoveToParentViewController:self];
    _bar.items = itms;    
    _bar.selectedItem = [itms objectAtIndex:0];
    _bar.autoresizingMask = UIViewAutoresizingFlexibleWidth;
    UIViewController *firstVC = [_viewControllers objectAtIndex:0];
    firstVC.view.frame = [self subviewFrame];
    firstVC.view.autoresizingMask = UIViewAutoresizingFlexibleWidth | UIViewAutoresizingFlexibleHeight;
    [self.view addSubview:firstVC.view];
-(void) viewDidLoad { = [[UITabBar alloc] initWithFrame:CGRectMake(0, 0, self.view.frame.size.width, 30) ];
    for(UIViewController *vc in _viewControllers) {
        [self addChildViewController:vc];
    _bar.delegate = self;
    [self.view addSubview:_bar];
    [self reloadBarInfo];
-(void) tabBar:(UITabBar *)tabBar didSelectItem:(UITabBarItem *)item {
    int idx = [_bar.items indexOfObject:item];
    UIViewController *newVC = [_viewControllers objectAtIndex:idx];
    UIViewController *curVC = [_viewControllers objectAtIndex:_selectedIndex];
    if(newVC == curVC) {
        return; // Don't switch - same one selected!
    [curVC.view removeFromSuperview];
    newVC.view.frame = [self subviewFrame];
    newVC.view.autoresizingMask = UIViewAutoresizingFlexibleWidth | UIViewAutoresizingFlexibleHeight;
    [self.view addSubview:newVC.view];
    [self transitionFromViewController:curVC toViewController:newVC duration:0 options:0 animations:nil completion:nil];
     self.selectedIndex = idx;

And there you have it!