FreeMarker Blog

The official weblog of the FreeMarker project

Sunday, February 14, 2010

FreeMarker on Google App Engine

We had several reports of issues in the past with use of FreeMarker on Google App Engine (GAE).

Analysis of these problems confirmed that runtime named "Java" in GAE is not, in fact, a Java-compliant runtime even though it's advertised as such, and looks like one at first sight. It doesn't provide access to some mandatory Java packages, and its reflection implementation has bugs; both of which issues affect FreeMarker.

We wish to stress that FreeMarker is being used in countless Java runtimes, many of them enterprise runtimes with strict security lockdowns, and it works in all of them, as long as these runtimes are conforming Java Runtime Environments. It is our belief that GAE is, unfortunately, non-conforming. Update: I have now reported the reflection bug in the GAE issue tracker.

This unfortunately doesn't help our users who would love to use FreeMarker on GAE. We can not quickly patch FreeMarker as some of the required workarounds would break backwards compatibility within the existing 2.3.x series, which is against our release policy; backwards compatible changes are only allowed when we increment first or second version numbers, that is, from current perspective, either in 2.4 or 3.0.

For this reason, I spent some of my weekend on this particular yak shaving, and have created a new version of FreeMarker's 2.3.16 JAR file that should run on GAE. I'm saying "should" because I don't use GAE myself, so it's up to you, the GAE users to verify it.

If you want to, please go to our project file releases, and grab "freemarker-gae-pre1.jar" from freemarker/2.3.16 directory. The file is labelled "pre1" as it's considered a prerelease; all you people who wish to use FreeMarker on GAE, please start using it and hammer at it, and report back any problems you can find. That said, I sincerely hope there will be none. The changes themselves are fairly light: I removed the dependency on Swing from AST classes, and worked around GAE's reflection security bugs. (If you see a different named file, it means we've updated it since, and please grab that more recent one.)

Update: There is now indeed a "freemarker-gae-pre3.jar"

This doesn't replace our longer term effort of having default FreeMarker be able to run under GAE, but is intended to help our users with an instant solution. There are no downsides to using this version instead of the official 2.3.16 release of FreeMarker, except for the backwards-incompatible change where the TemplateElement class no longer implements Swing TreeNode, but that incompatibility is probably exactly what you want under GAE. All other changes are under the hood - we had to expose few previously package-private implementation classes as being public to work around GAE's reflection security bugs. Since these classes are not documented in FreeMarker official JavaDoc anyway, they don't form the public API, so you shouldn't be using them even when you see them.

23 Comments:

At Tue Feb 16, 06:47:00 PM GMT+1, Anonymous Anonymous said...

Hi, thanks for publishing the GAE fix.

I am still getting an error:

HTTP ERROR: 500

freemarker.core.TextBlock.getParent()Ljavax/swing/tree/TreeNode;
RequestURI=/servlets/gadgetServlet

Caused by:

java.lang.NoSuchMethodError: freemarker.core.TextBlock.getParent()Ljavax/swing/tree/TreeNode;
at freemarker.core.TextBlock.isIgnorable(TextBlock.java:375)
at freemarker.core.TemplateElement.postParseCleanup(TemplateElement.java:240)
at freemarker.core.MixedContent.postParseCleanup(MixedContent.java:76)
at freemarker.core.FMParser.Root(FMParser.java:2961)
at freemarker.template.Template.(Template.java:168)
at freemarker.cache.TemplateCache.loadTemplate(TemplateCache.java:448)
at freemarker.cache.TemplateCache.getTemplate(TemplateCache.java:361)
at freemarker.cache.TemplateCache.getTemplate(TemplateCache.java:235)

 
At Tue Feb 16, 07:07:00 PM GMT+1, Blogger Attila Szegedi said...

Ah, the joys of incremental compilation - TextBlock wasn't recompiled... Sorry folks, should've known better than to not do an "ant clean" before building the JAR file. There's now a freemarker-gae-pre2.jar uploaded.

 
At Tue Feb 16, 07:20:00 PM GMT+1, Anonymous Anonymous said...

Seems to be working, thank you for your super fast response :)

 
At Tue Feb 16, 10:57:00 PM GMT+1, Blogger Unknown said...

What reflection bugs are you referring to? Do you have bugs filed in Google App Engine's public issue tracker?

 
At Wed Feb 17, 01:58:00 AM GMT+1, Anonymous Pierre said...

Works fine on GAE now !!

Thanks

 
At Wed Feb 17, 02:07:00 AM GMT+1, Anonymous Pierre (Lutece project) said...

Great Fix !

Thanks

 
At Wed Feb 17, 07:33:00 AM GMT+1, Blogger Attila Szegedi said...

Toby,

GAE will replace java.lang.reflect.* classes on the fly with its own doppelgangers defined in "com.google.apphosting.runtime.security.shared.intercept.java.lang.reflect." This causes problems when, say, a class tries to reflectively access a private member of itself, or a package-private member of another class in its package. This works in a compliant JRE as java.lang.reflect.* classes aren't subject to security checks, but it fails with GAE as their doppelgangers won't have the necessary permissions. See this message for an example:

http://www.mail-archive.com/google-appengine-java@googlegroups.com/msg00865.html

Here, a class named "freemarker.ext.beans.BeansWrapper" tries to reflectively create an instance of "freemarker.ext.beans.EnumModels" using a package-private constructor. In a compliant JRE, this works flawlessly, since they're both in the same package. Under GAE, it won't work because of intervening doppelganger Constructor instance. I'm not sure what GAE is trying to achieve by intercepting reflection (that couldn't be achieved by a carefully tuned security policy file instead).

 
At Wed Feb 17, 09:18:00 AM GMT+1, Blogger Attila Szegedi said...

Toby,

I have now filed an issue:

http://code.google.com/p/googleappengine/issues/detail?id=2790

 
At Wed Feb 17, 06:56:00 PM GMT+1, Blogger Unknown said...

Thanks for filing the issue, Attila. We'll look into it. As near as I can tell, there's nothing endemic here - just a potential bug in one reflective method under one specific circumstance.

The other bugs that you referenced are unrelated and have been fixed for some time now. (Now marked as so).

With respect to reflection and classloaders, GAE's security model is much more nuanced than tweaking a policy file and can not be expressed that way. In broad strokes, your application has permissions to do most "dangerous" reflective operations as long as it targets its own types. For example, you may read/write the private fields of your own classes but you may not do so to classes which belong to the JRE.

We strive to provide a very secure environment to prevent malicious applications from attacking your application or Google's infrastructure. On the other hand, we go to great lengths to give applications access to reflective and classloading operations that are normally not available in a sandboxed environment due to their "unsafeness".

 
At Wed Feb 17, 07:11:00 PM GMT+1, Blogger Attila Szegedi said...

Never said it's endemic, just that it prevented folks from using FreeMarker on GAE.

The problem was exactly in accessing package-private members of application specific classes.

Anyway, I appreciate the communicativeness and the discussion on your part; our temporary workaround is implemented by exposing some formerly package-private classes as public; if we can reverse that at some later point, I won't mind it :-)

Thanks again for chiming in.

 
At Fri Mar 26, 01:20:00 PM GMT+1, Anonymous SN said...

Thanks for fixing this up.
This is what I was looking for.

 
At Tue Apr 27, 12:06:00 PM GMT+2, Anonymous SN said...

I am using freemarker-gae-pre2.jar and I get following error when I deploy to GAE. Its working fine on my local machine
I am using spring MVC + freemarker.

Error -

java.lang.VerifyError: (class: freemarker/ext/jsp/FreeMarkerJspApplicationContext, method: signature: ()V) Incompatible argument to function

 
At Tue Apr 27, 03:14:00 PM GMT+2, Blogger Phuong Nguyen said...

Thank for the fix. But, there is one more error when I'm trying to render a view (I'm using FreeMarkerServlet in accordance with Apache Tiles 2, though):

java.lang.NoClassDefFoundError: Could not initialize class freemarker.ext.jsp.PageContextFactory
at freemarker.ext.jsp.TagTransformModel.getWriter(TagTransformModel.java:99)
at freemarker.core.Environment.visit(Environment.java:286)
at freemarker.core.UnifiedCall.accept(UnifiedCall.java:130)
at freemarker.core.Environment.visit(Environment.java:210)
at freemarker.core.MixedContent.accept(MixedContent.java:92)
at freemarker.core.Environment.visit(Environment.java:210)
at freemarker.core.Environment.process(Environment.java:190)
at freemarker.template.Template.process(Template.java:256)
at freemarker.ext.servlet.FreemarkerServlet.process(FreemarkerServlet.java:452)
at freemarker.ext.servlet.FreemarkerServlet.doGet(FreemarkerServlet.java:391)

Would you join us by building a sample app that utilize FreeMarker and test it on GAE? FreeMarker is cool and we have adapted it quite deeply. It would be a nightmare to not have it running properly on GAE.

 
At Tue Apr 27, 03:39:00 PM GMT+2, Blogger Attila Szegedi said...

I believe NoClassDefFoundError might be a consequence of the VerifyError; it's the way JVM works -- if loading of a class fails for whatever reason (including failure to load another class it depends on during loading), then JVM will remember that it failed to load the class, and on next attempt, it will quickly fail with NoClassDefFoundError instead.

So, I'd ignore this error for a while and just go and find the first error that occurred before the NoClassDefFoundError - that is the real cause of your problem.

 
At Tue Apr 27, 07:00:00 PM GMT+2, Blogger Phuong Nguyen said...

I have deployed a different version and NoClassDefFound is the only error I can see. No VerifyError or any other error there, and I'm invoking the request for the first time. I believe 2 different versions of my webapp should be run in 2 different JVMs and thus would not share any loaded class. By the way, I have waited for 4 hours before trying again. Same result.

 
At Wed Apr 28, 04:31:00 AM GMT+2, Blogger Phuong Nguyen said...

Oh, sorry for wrong information. I have tried again after 9 hours and here is the real exception:

java.lang.VerifyError: (class: freemarker/ext/jsp/FreeMarkerJspApplicationContext, method: signature: ()V) Incompatible argument to function

Same with before. Wonder why 2 different versions doesn't have separated JVM.

 
At Wed Apr 28, 07:44:00 AM GMT+2, Anonymous SN said...

Whoever want's above issue to be fixed should vote here http://code.google.com/p/googleappengine/issues/detail?id=3157

 
At Mon May 03, 08:02:00 AM GMT+2, Anonymous sn said...

Attila, Any luck with above issue?
Please do some thing needful. Freemarker community needs it to run on GAE

 
At Mon May 03, 08:59:00 AM GMT+2, Blogger Attila Szegedi said...

I'm sorry, but I just don't see what could I do. This code has been working without problems for the last 7 years in countless Java runtimes. I don't see anything problematic with it, therefore I don't know what would need to be changed. I do not know how to work around it. I'll contact a Google Developer Advocate I know, maybe he can help us to get someone on Google's end to look into this.

 
At Mon May 03, 03:38:00 PM GMT+2, Anonymous sn said...

Thanks Attila,

Just wanted to know one more thing.
How should I take freemarker regarding the GAE compatibility. right now we are in the process of setting up a new GAE+Java project. Should we consider Freemarker !.

Can you tell us, does freemarker team has any future plan to make the freemarker compatible to GAE !. That mean, officially supporting GAE as a platform on which freemarker can run.

The GAE Pre release is nice, But we can't decide just but based on that, weather we should choose freemarker or not.

The current state of freemarker regarding GAE compatibility indicates that freemarker isn't a choice for GAE.

You answer will help many taking the right decision.

Thanks
SN

 
At Sun May 23, 09:37:00 AM GMT+2, Anonymous sn said...

Attila,
Freemarker + Sitemesh jsp decorators + GAE does not work.

Do you have any solution for
http://code.google.com/p/googleappengine/issues/detail?id=3252

 
At Wed Jun 16, 11:15:00 AM GMT+2, Anonymous Anonymous said...

There also seems to be an issue with using JSP Tags from freemarker on AppEngine (using pre3).

Uncaught exception from servlet
java.lang.IllegalAccessError: Class com.google.apphosting.runtime.security.shared.intercept.java.lang.reflect.Method_$1 can not access a member of class freemarker.ext.jsp.FreeMarkerPageContext21 with modifiers "static"
at freemarker.ext.jsp.PageContextFactory.getCurrentPageContext(PageContextFactory.java:60)
at freemarker.ext.jsp.TagTransformModel.getWriter(TagTransformModel.java:100)
at freemarker.core.Environment.visit(Environment.java:285)
at freemarker.core.UnifiedCall.accept(UnifiedCall.java:130)
at freemarker.core.Environment.visit(Environment.java:209)
at freemarker.core.MixedContent.accept(MixedContent.java:92)
at freemarker.core.Environment.visit(Environment.java:209)
at freemarker.core.Environment.process(Environment.java:189)
at freemarker.template.Template.process(Template.java:237)

 
At Sat Nov 13, 09:20:00 PM GMT+1, Blogger Gray said...

You might want to put an update at the top of this blog entry because people still think that FreeMarker does not work with GAE while it works fine for me.

 

Post a Comment

<< Home