Thursday, May 6, 2010

Converting Google Protocol Buffers To JSON Objects and vice versa in Java

I am working on an application and experimenting with Google Protocol Buffers.  The idea is the back end service communication will use Protocol Buffers but I have some components which will be accessed by a service on the web.

I searched around on Google and found a few references from Google engineers stating that it would not be difficult to convert Protocol Buffers to JSON or the other way around as well as references hinting that this is how Google Maps works but I could not find any code for it.

I did find a JavaScript library that is supposed to encode and decode the binary format but it did not really make sense to me to go this route on the front end since the HTTP request still has a lot of text in it and the savings of binary encoding for most HTTP requests seems minimal.  Anyway JavaScript does not really seem to be suited for reading and writing binary protocols.  Add to that the fact that the library did not even have solid release and it was time to roll my own converter.

Mainly I just wanted to reuse the object models I create in my .proto files on the front end with those I use on the back end and this was the best way I could think of to do this.  I have included below the code I came up with which works for my needs.  I did not implement for all cases but if it is useful to your project the code is all there to improve on.

I should mention this uses a library for Java from "JSON.org".  The models were just generated from my .proto files using protoc.  

Anyway without further ado here is the code.  If you make any improvements on this or have ideas on how to improve it please let me know.  One other word of explanation the protocol class here if for my application the important part is the "decodeMessage" and "encodeMessage" part so I have removed some of the irrelevant parts from this.


package hak.server3.protocol;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

import com.google.protobuf.Message;
import com.google.protobuf.Descriptors.Descriptor;
import com.google.protobuf.Descriptors.FieldDescriptor;

import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;
import java.util.Map;

import org.json.JSONException;
import org.json.JSONObject;

import hak.server3.Protocol;

/**
 * This protocol reads and writes JSON it is primarily meant for communication
 * between browsers and internal services.
 * 
 * @author ben hakala
 */
public class JSONProtocol implements Protocol {
// This is from JSONObject but is not visible os I am putting it here
static boolean isStandardProperty(Class clazz) {
        return clazz.isPrimitive()                  ||
            clazz.isAssignableFrom(Byte.class)      ||
            clazz.isAssignableFrom(Short.class)     ||
            clazz.isAssignableFrom(Integer.class)   ||
            clazz.isAssignableFrom(Long.class)      ||
            clazz.isAssignableFrom(Float.class)     ||
            clazz.isAssignableFrom(Double.class)    ||
            clazz.isAssignableFrom(Character.class) ||
            clazz.isAssignableFrom(String.class)    ||
            clazz.isAssignableFrom(Boolean.class);
    }
public static Message decodeMessage(JSONObject obj) throws JSONException {
String type = obj.getString("_pb_class");
if(type == null) {
throw new JSONException("Cannot decode message, missing _pb_class");
}
Message msg = null;
try {
// Create an instance of this class and set up its properties
Class cls = Class.forName(type);
Method m = cls.getDeclaredMethod("newBuilder", (Class[])null);
Object builder = (Object) m.invoke(null, null);
// now go ahead and put this message together
Iterator iter = obj.keys();
Class bcls = builder.getClass();
Method[] methods = bcls.getDeclaredMethods();
while(iter.hasNext()) {
String key = iter.next();
if( ! key.equals("_pb_class")) {
String methnm = "set";
String[] parts = key.split("_");
for(int i=0; i < parts.length; i++) {
if(parts[i].length() > 0) {
//System.out.println("parts["+i+"] is '"+parts[i]+"'");
String fc = parts[i].substring(0,1).toUpperCase();
parts[i] = parts[i].toLowerCase().substring(1,parts[i].length());
methnm += fc+parts[i];
}
}
//System.out.println("Looking for method "+methnm);
m = null;
for(int c=0;c < methods.length;c++) {
Method m2 = methods[c];
//System.out.println("Checking method "+m2.getName());
if(m2.getName().equals(methnm)) {
//System.out.println("Found method "+m2.getName());
m = m2;
break;
}
}
m.invoke(builder, obj.get(key));
}
}
msg = ((Builder) builder).build();
} catch(ClassNotFoundException e0) {
throw new RuntimeException("Failed to decode message.", e0);
} catch(IllegalArgumentException e1) {
throw new RuntimeException("Failed to decode message.", e1);
} catch(IllegalAccessException e2) {
throw new RuntimeException("Failed to decode message.", e2);
} catch(InvocationTargetException e3) {
throw new RuntimeException("Failed to decode message.", e3);
} catch(SecurityException e4) {
throw new RuntimeException("Failed to decode message.", e4);
} catch(NoSuchMethodException e5) {
throw new RuntimeException("Failed to decode message.", e5);
}
return msg;
}
public static JSONObject encodeMessage(Message message) throws JSONException {
JSONObject obj = new JSONObject();
obj.put("_pb_class", message.getClass().getName());
Classextends Message> mclass = message.getClass();
Descriptor d = message.getDescriptorForType();
List fs = d.getFields();
for(FieldDescriptor f : fs) {
String key = f.getName();
String methnm = "get";
String[] parts = f.getName().split("_");
for(int i=0; i < parts.length; i++) {
String fc = parts[i].substring(0,1).toUpperCase();
parts[i] = parts[i].toLowerCase().substring(1,parts[i].length());
methnm += fc+parts[i];
}
Object result = null;
try {
Method m = mclass.getDeclaredMethod(methnm, (Class[])null);
result = m.invoke(message, (Object[])null);
} catch (Exception e) {
            throw new RuntimeException(e);
}
//System.out.println(">> FIELD: "+f.getName()+" "+f.getType()+" "+methnm+" "+result);
if (result == null) {
            obj.put(key, (String)null);
            } else if (result instanceof Message) {
            obj.put(key, encodeMessage((Message)result));
            } else if (result.getClass().isArray()) {
            throw new RuntimeException("Not implemented yet");
            //obj.put(key, new JSONArray(result, false));
            } else if (result instanceof Collection) { // List or Set
            throw new RuntimeException("Not implemented yet");
            //obj.put(key, new JSONArray((Collection)result, false));
            } else if (result instanceof Map) {
            throw new RuntimeException("Not implemented yet");
            //obj.put(key, new JSONObject((Map)result, includeSuperClass));
            } else if (isStandardProperty(result.getClass())) { // Primitives, String and Wrapper
            obj.put(key, result);
            } else {
                if (result.getClass().getPackage().getName().startsWith("java") ||
                        result.getClass().getClassLoader() == null) {
                obj.put(key, result.toString());
                } else { // User defined Objects
                throw new RuntimeException("Not implemented yet");
                    //obj.put(key, new JSONObject(result, includeSuperClass));
                }
            }
}
return obj;
}
}

Tuesday, February 9, 2010

I was playing around with creating Jquery plugins a while ago.  I wanted to comment on this for others.  It turns out making plugins is rather easy.  Jquery is just the coolest javascript library I have ever seen.  There are a number of good templates online and grabbing a simple template and extending it is the easiest thing to do.  I found this one by Dave Smith and you should also refer to the Official Docs.  The reason I like a simple template to start with like the one Dave has is sometimes the official docs since they have to cover every options do not explain how to get started very well.

One comment on Daves template is that one change should be to make the functions that all functions that are not "get" functions should return a reference to "this" so you can chain function (see example).

So basically here is my (slightly modified template):


(function($)
{
$.fn.plugin = function(options) {
  // Set the options.
  options = $.extend({}, $.fn.plugin.defaults, options);
  // Go through the matched elements and return the jQuery object.
  return this.each(function()
  {
  });
};
// Public defaults.
$.fn.plugin.defaults = {
property: 'value'
};
// Private functions.
function func()
{
return;
};
// Public functions.
$.fn.plugin.func = function()
{
return this;
};
})(jQuery);


Now for an example.  Say we wanted to make a trivial template that sets all elements using this plugins color to red and enables you to change the font color and the background color.  You can take the template above and change it like this:



(function($) {  
$.fn.mycolor = function(options)  {   
// Set the options.   
options = $.extend({}, $.fn.plugin.defaults, options);    

// Go through the matched elements and return the jQuery object.   
return this.each(function()   {
// do something with this. Say we wanted to make it red
this.css("color", options.color);
});  
};
// Public defaults.  
$.fn.mycolor.defaults = { 
color: 'red' 
};
// Public functions.  
$.fn.mycolor.setColor = function(color)  {
this.css("color", color);   
return this;  
}; 

$.fn.mycolor.setBackgroundColor = function()  {
this.css("background-color", color);   
return this;  
};
})(jQuery);



Now you can do this:


$(".color_example").mycolor();


This will make all html elements with the class .color_example change colors to red, then to change their colors later in some other context you can do this:


$(".color_example").setColor("blue").setBackgroundColor("black");

Note that unless setColor returned "this" you would have to do this in two function calls.  This is so stupidly trivial it is not useful but hopefully this helps get the idea of how easy it is to make a plugin and do cool things with it.

Tuesday, February 2, 2010

Over the couple years I have been using Zend Framework to an increasing degree in projects I have been working on.  At first I just used a small set of Zend Frameworks features.  For example some other developers and I liked the way the form validation was set up so we started using Zend validators in our forms, we used Zend_Db_Table, Zend_Acl,  and other various components.  Over time we started using the full fledged Zend Framework MVC system enabling us to slowly weed out various proprietary libraries we had developed over time.

For the most part my experience with Zend Framework has been positive.  The framework is well documented relatively easy to use, is well tested and designed but there is one aspect of Zend Framework which seems very non-intuitive and confusing to work with, namely Form Decorators.

Form Decorators seem like a very cool idea in theory.  The idea of defining a form as a sort of specification that can be rendered dynamically in such a way that the view of the form is abstracted from the specification opens some cool doors.  For example we can start to think of every form element now as a basic Object which can be reused in many forms on many pages within a given site and even reused across multiple sites.  The validation for the element is tied to the element and is therefore inherited in each form it exists within.

Consider some common form elements like address, postal code, email address, country, state, etc.  All such elements with some minor variation are basically the same in most forms you would put them in.  Country and States usually need to be populated from a database so the element can also contain the logic for fetching the list from the database and populating it's internal array of options.  States are usually contextual to the country so you need to add a trigger to the the country element to update the state list when a country is selected.

I have been fairly successful tying this together with JavaScript and placing all of the logic and configurations for all of these kinds of things with my element classes.  My favourite library for client side JavaScript is Jquery which in my opinion is one of the coolest things to happen to JavaScript ever.  When I think back to the browser war where Microsoft was trying to snuff out Netscape.  We used to have to create all these libraries to abstract basic functions so they would work in entirely different ways depending on the platform, then a new version would come out and break it.

The W3C standards that are supported by all modern browsers has helped make it possible to write portable JavaScript code without the need for such libraries but Jquery is not so much a wrapper library to translate code to various platforms like we did in the old days it is a new way of thinking about JavaScript, it is a way of doing a lot with a few lines of code.  The difference between writing code in plain JavaScript and with Jquery is sort of like the difference between writing procedural code to look for patterns in textual strings vs. using regular expressions.  It is just far cooler and more efficient.

Anyway enough on that subject back to Zend Forms and decorators.  So hopefully the idea is clear.  Form elements can contain a common and reusable payload which defines attributes of such elements and forms are like a bundle of such elements.  Zend_Form contains various options for creating groups of elements wrapped in a fieldset and such forth but like I said the one thing that seems a bit counter intuitive is how decorators work.

At first I would google for "Zend_Form decorators table" for instance and find examples on how to render a form in a table or whatever but the problem is that while many examples exist it is always a pain to find one that actually does exactly what you want.  Furthermore without some understanding of fundamentally how they work it is difficult to modify them to do what you want.  Even if you find an example that is close to what you want it is never exactly the same so some modification is almost always required.

Some developers recommend just bypassing the form rendering methods built in to Zend_Form and just building custom views for each form and just dynamically rendering the form elements and labels manually in the template because it is so weirdly complicated and annoying to deal with.  The problem with this is you have now thrown away all of the coolness and abstraction that has been carefully set up in Zend_Form.  Now a form is tied to a view and you cannot change one without also changing the other.

It seems to me if you have to manage the code in more then one place you might as well not use Zend Form.  Just do it the old fashioned way and make a page with hard-coded HTML form elements and write code to handle the post variables that come in.  I guess I can think of some exceptions where Zend_Form coupled with customized views makes sense but for the most part it seems like one ought to just be able to work with decorators and make the form render the way you want.

In any case this post is mostly meant to phrase the problem and put it out there.  Over the next week I plan on looking over the Zend Framework source code and see if I can make some sense of how to use decorators.  The various how to's and forum entries on the web answer only part of the question of how to use decorators and only usually within some context which does not give you a complete picture.  I will try to follow up when I have time hopefully with a clear answer of how the heck they are supposed to work.

Friday, January 29, 2010

Disabling foreign key constraints so you can delete records in Oracle

From time to time in Oracle we need to delete some records from a table which is referenced by foreign keys. Sometimes if you delete the records that refer to the records you are trying to delete you can delete them (if no records refer to those records primary keys). Other times even if no records refer to the records you are trying to delete you still will not be able to delete them just because of the presence of the foreign key. 

The workaround is after you verify that there are no records referring to the records you want to delete you can temporarily disable the foreign key constraint then delete your records then enable the constraints. To find the constraints run the following query after changing 'SOME_TABLE' to your table name:








select 'alter table '||a.owner||'.'||a.table_name||
' disable constraint '||a.constraint_name||'
/'
from all_constraints a, all_constraints b
where a.constraint_type = 'R'
and a.r_constraint_name = b.constraint_name
and a.r_owner = b.owner
and b.table_name = 'SOME_TABLE'
/









You can also specify the schema I usually just check the output before running for tables with the same name in the same schema.

If you then copy the sql generated from this query you then run it to disable the constraints. Then run you deletes. Afterward just change the keyword "disable" in the statements and change them to "enable" and run that to enable the constraints.

If you were correct that there were not records referencing the records through the foreign key the constraint will enable successfully. If it doesn't then you just need to find the corresponding records and remove them.


For example if you ran the above query and 4 tables had foreign keys to 'SOME_TABLE' then the above query might generate the following:









alter table SOME_SCHEMA.TABLE1 disable constraint SOME_FK
alter table SOME_SCHEMA.TABLE2 disable constraint SOME_OTHER_FK
         
alter table SOME_SCHEMA.TABLE3 disable constraint SOME_OTHER2_FK
 
alter table SOME_SCHEMA.TABLE4 disable constraint SOME_OTHER3_FK
/  

Run this then you will be able to successfully execute:

delete from SOME_SCHEMA.SOME_TABLE where ...
 





Then change disable to enable and run that:

alter table SOME_SCHEMA.TABLE1 enable constraint SOME_FK
alter table SOME_SCHEMA.TABLE2 enable constraint SOME_OTHER_FK
         
alter table SOME_SCHEMA.TABLE3 enable constraint SOME_OTHER2_FK
 
alter table SOME_SCHEMA.TABLE4 enable constraint SOME_OTHER3_FK
/  
and your records are gone and your constraints are back up.  Just make sure you remove the records referring to the records you are trying to delete first!  Sometimes those records may not be deleted until some other constraints are disabled.  Follow the dependencies as deep as needed repeating this process for each table you can't delete from.

Wednesday, January 27, 2010

Go Proverb - Always look to the strongest cut

Do not miss the chance to cut when your cutting stone cannot be captured and the cut is advantageous (and preferably sente).

Tuesday, January 26, 2010

Mac OS X Enable Debug Menu Safari

I had to reinstall OS X the other day and I need to enable the debug menu again in Safari and I always forget how the heck to do it so I thought I would post it.

The trick is:
  1. Close Safari
  2. Open Terminal
  3. Type the following command in Terminal: defaults write com.apple.Safari IncludeDebugMenu 1
  4. Launch Safari.
You should now see a new menu called Develop on the Safari menu. This is a very cool feature for developers. Inspect element is a nice feature and it has a lot of nice script debugging features.