Daniel Nouri's Blog - Latest Comments

Re: How to write an add-on for the Kotti CMS — Daniel Nouri's Blog

Daniel Nouri — Fri, 23 Mar 2012 08:52:13 -0000

The source code is at https://github.com/Pylons/Kott...

Re: How to write an add-on for the Kotti CMS — Daniel Nouri's Blog

Progoney — Fri, 23 Mar 2012 00:57:35 -0000

where's the source code?

Re: How to write an add-on for the Kotti CMS — Daniel Nouri's Blog

Simon — Fri, 10 Feb 2012 22:11:03 -0000

I like the design of kotti. Plain and simple as it should be !

Re: Daniel Nouri's Blog

pgollakota — Fri, 18 Nov 2011 06:05:38 -0000

Daniel, If you are using django and highcharts together for your projects, you may find a new django app that I wrote called django-chartit helpful. http://chartit.shutupandship.c.... Thanks!

Re: Daniel Nouri's Blog

dnouri — Wed, 01 Jun 2011 05:26:25 -0000

Here's a pretty nice explanation of how ESI works and why it's better than caching whole pages:
http://www.trygve-lie.com/blog...

Re: Daniel Nouri's Blog

Jean — Wed, 23 Mar 2011 02:21:01 -0000

If I'm reading it on my RSS reader, that usually means I'm it reading offline on my phone in a bus.

Re: Daniel Nouri's Blog

Douglas Soares de Andrade — Fri, 18 Mar 2011 14:15:08 -0000

Congrats Guys ! I really would love to work on something like that.

Re: Daniel Nouri's Blog

valentim — Fri, 18 Mar 2011 08:13:54 -0000

I want to thank Daniel for his post and his role on this project. His help was, and still is, very important for us! But it was also his fault that he gave such a good training!
We really do like Pyramid and are quite happy to see that we did bet on the winning horse (Pylons, TurboGears2 and Repoze.bfg ended up merging and created this amazing framework).
Pyramid is easy, extensible, stable and well organized.

To give an overview of the most important Pyramid based apps we built for this project, check the items below.

# Profile app
This is one of the most important applications. It gathers information from several sources using Webservices (SOAP/JSON) and database mapping, and allows the user to have access to all of his personal information, as well as tools to manage his information. Some of the information displayed is employee receipts, work time statistics, email and wireless statistics, lists of absences from work and several tools to manage his email account.
It actually grabs info from SAP, LDAP directory and mail system, processes the data and then we can present all this in a cool UI.

# Accounts Management app - LDAP Alchemy?
This is a Python application created to manage email/LDAP accounts. For this we wrote a Python-Ldap API to deal with all the LDAP interaction and to create an abstraction layer so that we could easily integrate LDAP into permissions mapping, logging, etc. Works like SQLAlchmey, but for LDAP :) We are thinking of publishing the package/library as opensource but because it was developed in a employer/employee work environment we don't know if we can make it avaliable.
We also created SOAP webservices (based on soaplib) for external applications to consume.

# My Page app
Mypage is great! We love it! You can build your own personal page very easily and quickly, needing only to have basic internet skills. Any UC account can use this application and start their institutional webpage with just one click. The users can manage permissions to each of their pages and also have bilingual support.
You can check my homepage with dummy info as a demo: http://staff.uc.pt/valentim

# HelpDesk app
This is another example of the integration efforts we put into this project. Our helpdesk system is based on BestPractical's RT (Request Tracker) and we built a Python API to work as an abstraction layer and communicate via REST with RT. This application also has a webinterface to let users access their requests status, create tickets, respond to tickets, etc without the need to access RT which is not very user friendly or pleasing to the eye.

# Subscriptions app
Here at UC we have a lot of mailing lists and newsletter offers for students/teachers/etc and they are based on Mailman. It's difficult for any user to know about all of these offers and also to manage their subscriptions. This applications creates an interface between Mailman and the user, allowing them to easily check all the offers and subscribe/unsubscribe from them.

# Go! app
It's a basic redirect system that allows users to create shortcuts for applications/resources/long url addresses. Completely written in Python it integrates an approval system so that anyone can suggest new shortcuts while keeping things "safe". For example www.uc.pt/go/wifi -> www.uc.pt/ciuc/servicos/eduroa... (which is the information page about the UC wireless network).
You can check this app, is public: http://apps.uc.pt/go

# Docs - KnowlegdeTree integration
This apps is an integration of the KnowlegdeTree DMS using it's SOAP interface. At this moment it's only an interface to access the documents and folders, with no write or upload for now. We'll get there!

# Logging
All our apps are connected using this logging library we built. It allows us to store necessary data and have an overview of what's happening with our apps without the need for an ugly terminal.

# User Interface
The layout was designed to reflect this integration work as much as possible. After authentication we want to keep the user a click away from all the applications they can access. For this we have a drop down menu that works very much like Android's notification area, but with application shortcut icons. Trying to follow the wave, we built the user interface to work also in a tablet environment. This interface was designed to work in both the public area as well as the management applications. We are using HTML5 and CSS3 and Javascripting.

You can also check out this video we made for the project: http://vimeo.com/19594790 .
Sorry, it's Portuguese only!

Re: Daniel Nouri's Blog

Daniel Nouri — Fri, 18 Mar 2011 07:49:48 -0000

You're right. What I'm doing here with "function node_changed" and the "node_changed_handlers" array is implement my own subscriber pattern. I'll consider using a custom event the way you describe it in Kotti. But will leave the example here to use an array.

Re: Daniel Nouri's Blog

valentim — Fri, 18 Mar 2011 07:45:59 -0000

Nop! We are using Silva in a separated virtualhost, but the authentication is shared, using shibboleth.
We made two plugins for repoze.who: shibboleth and a json plugin.
From shibboleth we get the userid, the with json we get all the other user metadata, from a REST JSON response running in the main server (connected to ldap).

Re: Daniel Nouri's Blog

Maurizio Lupo — Fri, 18 Mar 2011 04:32:44 -0000

I think your solution should be applied to other framework too (Plone for example :-) ).

A more elegant approach can use a custom event (jquery can manage these). Like this:
$('html').bind('loadsection', function (evt){
$node_changed($(evt.target));
});

...
and after loading a part of html
$('#just-modified').trigger('loadsection');

Re: Daniel Nouri's Blog

wik — Thu, 17 Mar 2011 18:35:32 -0000

Still they using Silva as one of those apps? :)

Re: Daniel Nouri's Blog

Daniel Nouri — Sun, 13 Mar 2011 14:24:51 -0000

Fixed the link. Thanks.

Re: Daniel Nouri's Blog

Maxm — Sun, 13 Mar 2011 11:32:43 -0000

(If you're looking at this through your RSS reader, you might want to head over to my blog for some synyax highlighting.)

hehe in my rss reader it leads to localhost

Re: Daniel Nouri's Blog

Daniel Nouri — Thu, 17 Feb 2011 12:07:39 -0000

I should correct myself and say that "purge all on edit" _is_ faster than assembly when indeed all parts of all pages changed. It's faster then because of the overhead of additional requests to the application server. It's just not a very likely scenario.

Re: Daniel Nouri's Blog

Daniel Nouri — Wed, 16 Feb 2011 10:29:01 -0000

Hey Laurence.

I disagree with your assumption that it's "clearly faster to deliver a page without having to execute any includes". Why? Because with dependency tracking, your page needs to be retrieved from your app server much more often.

The "purge all on edit" strategy that you present is simple, but it's never faster than ESI and suitable only for sites up to a certain size/amount of traffic. And Varnish is super fast with doing the actual assembly work when all parts can be retrieved from the cache.

(I'm repeating myself just for clarity and for others who are reading this.) Why is page dependencies not faster? Because with page dependencies, the lifetime of your page becomes that of the page element with the shortest lifetime. Let's imagine you have a static twitter sidebar that you want to refresh every five minutes. That'd mean that _all_ your pages have a maximum lifetime of five minutes. Or navigation: not all your site's content needs to be purged just because you add an <li> to your navigation, right? Here, it's not even _desirable_ to get the dependencies right; they wouldn't help.

Regarding "re-engineer Plone to use ESI": Plone's default skin in itself only works for public sites up to a certain size because it's really slow, that is, requests that _need_ to get through to your app server will decrease performance. (By the way, XDV- based sites suffer from the same problem since they'll be as slow as the Plone skin behind them.) For that reason, and for reasons of better control and ease of skinning, I suggest that for popular public sites, one should use a completely separate skin for the public view, at which point the Plone UI becomes an "admin UI". That's exactly what we did for www.vienna.info -- using collective.skinny [1]. With a separate skin it's fairly easy to serve Plone content with ESI.

Not saying it's desirable for every public Plone site out there to use ESI or a separate public skin. Just for those where speed matters a lot.

[1] http://pypi.python.org/pypi/co...
</li>

Re: Daniel Nouri's Blog

Laurence Rowe — Wed, 16 Feb 2011 09:29:04 -0000

When you have logged in users, then you must use ESIs for personalisation, but I'm not trying to solve caching for facebook here ;-)

The dependency tracking feature is intended for public sites, i.e. ones where nobody is logged in. For that situation it's clearly faster to deliver a page without having to execute any includes. (The dependencies are there for invalidation and Varnish has a thread that works through all objects in the cache comparing them against the ban list so the invalidation check never becomes problematic.) It also means you don't have to re-engineer Plone to use ESIs. Now, it's certainly not without problems - it can be very easy to flush large portions of your site from the cache, and the implementation is currently incomplete when you need to add a new source of information (i.e. a new portal tab.)

The simplest solution for small sites with heavy traffic and few updates is just to invalidate all html content in the cache whenever you make a page edit. This ensures you get consistency of navigation elements between pages while allowing all requests to normally get served from the cache. You can set this up with plone.app.caching by switching on 'Moderate caching' for content items (so they get cached) and tweaking your Varnish config to additionally invalidate all html pages on PURGE with:

purge("obj.http.Content-Type ~ ^text/html");

Re: Daniel Nouri's Blog

Daniel Nouri — Mon, 14 Feb 2011 18:53:06 -0000

I think the problem with pages with multiple dependencies is a misunderstanding. As I said before, it's both slower and more complex than using an alternative technique called Edge Side Includes. Here's why:

Take Facebook as an example: they don't calculate dependencies between pages -- that would kill their performance since there's always some bits of the page that vary.

Instead, what they do is split up the page into different parts and use Edge Side Includes to pull these parts together. What they gain is they can reuse most of the HTML between logged in users, and keep some parts cached longer than others. (ESI allows them to keep the friends list cached longer than the timeline.)

When you look at a photo album in Facebook, you're probably looking at a cached version of it. "But how is that possible, because I'm logged in?!" Since Facebook doesn't cache the whole HTML page starting from to , but rather small parts of it (say from <div> to </div>), they can still fetch most of the page from their Varnish cache before they assemble the page and hand it out to you.

We're using Edge Side Includes heavily in http://www.vienna.info . Parts like the login menu on the top right can't be cached when users are logged in since they will display the user's name. But most of the rest of the page (content area, navigation, sidebar) is cached in one way or another.

Sure, many Plone sites aren't big enough. For them it's good enough to use the default skin, maybe some XDV and plone.caching. But _if_ you're dealing with a large site and need more speed, you should definitely look into ESI, and not into calculating hard-to-get-right and inefficient page dependencies.

http://en.wikipedia.org/wiki/E...

Cheers
Daniel

Re: Daniel Nouri's Blog

Laurence Rowe — Sun, 13 Feb 2011 20:02:21 -0000

304's are useful for logged in traffic, where you can't cache page views in Plone as each page renders differently for each user (e.g. a standard intranet scenario.) For anonymous visitors (where you can cache pages in Varnish) then 304's only really help save bandwidth and visitors with fast connections won't notice.

The problem of invalidating cached pages with multiple dependencies is interesting. I had a go trying to make it work in http://dev.plone.org/collectiv..., though it was more of a thought experiment than anything else.

Re: Daniel Nouri's Blog

Daniel Nouri — Sun, 13 Feb 2011 05:14:16 -0000

For me, semantically, the application seems like the more natural place to put my caching rules. For one, there's a benefit in living close to your model in that you can introspect it in great detail before deciding the caching strategy for it. With Varnish you only see half the picture, and you'd need to pass on data from your app like with the X-Portal-Type header that you describe.

I disagree that it's more simple to test changes with Varnish. I've found the debugging and testing cycle to be much more pleasant with Python than with Varnish's VCL. That's why I put as much as is feasible into Python. Some parts, however, like some of the handling of ESI actually require you to hack the VCL. (That's true because Varnish doesn't respect the response cache headers of ESI parts.)

You're right in that, sometimes, you need the flexibility of running on every request. However, for the reasons mentioned, I like to keep those configurations in VCL as small as possible. You could probably argue that it's somewhat messy when you start putting your HTTP cache logic in two different places. But for me, Python just works much better than VCL.

I'd be interested see other examples of Varnish configuration where you need the flexibility of running on every request. Apart from ESI, I've found that to be true only for checking authentication headers.

Re: Daniel Nouri's Blog

Daniel Nouri — Sun, 13 Feb 2011 05:12:20 -0000

Put into perspective and compared with a cache hit in Varnish, a 304 *is* practically as slow as a full response. Instead of repeatedly asking the CMS if a content has changed or not, just let it expire periodically, and get it anew, say, every hour. I believe that most websites don't need validation, and it'll actually bog them down. But again, it's important to get the strategy right first, and the _choose_caching_policy that I've presented is just an example of an implementation of one such strategy. While the rest of the code and the general approach will work nicely with whatever strategy you come up with.

Regarding the complex dependencies that you mention that stem from changing navigation trees and the like: these dependencies are very hard to get right, and dangerous -- imagine you make a change to your site navigation, you'd have to invalidate all your HTML pages.

Avoid the "least common denominator" approach where you invalidate a whole page if only a small part of it have changed. Use Edge Side Includes instead. It's not only easier to grasp than trying to calculate dependencies, it's also much faster than invalidating whole pages. I've explained this approach at our talk at the PloneConf 2009 in Budapest.

In summary, I'm still arguing that people should prefer "roll your own" (and start by copying the 'cache' module) for higher traffic websites, since it'll give them the exact control they'll ultimately need.

Re: Daniel Nouri's Blog

optilude — Sat, 12 Feb 2011 19:34:47 -0000

When you know what you're doing, a combination of plone.app.caching and Varnish can be incredibly powerful. We ended up doing very clever things with custom headers, cookies and specific Varnish config to really squeeze performance out of a Plone site where many people were logged in (but not content authors). The end result is very fast, but also quite complex. To work with it requires a very good understanding of HTTP caching, Plone, our custom code, and the usage profiles for the site. I don't think you can get away from that when you need that level of optimisation.

Re: Daniel Nouri's Blog

optilude — Sat, 12 Feb 2011 19:31:54 -0000

Hi Daniel,

The rather extensive load tests we did on our last project says 304's make a huge difference. In plone.caching, they are significantly faster than rendering a full request in a Plone site, as all they do is traversal, then an O(1) lookup, then a response. We found a page that "leaked" through the load test without caching or 304s, and it killed performance at 50% of our target traffic. Once we plugged that hole, we sailed through at nearly 200%.

Caching content for 1 hour is fine if your site doesn't have complex dependencies. But it's not fine for most Plone sites where you will end up with e.g. a navtree for a folder where two pages show a different version of the same navtree (e.g. one was cached before an update that added a new child entry), or where a portlet added in a section will show up on some pages in that section, but not others (until the caches for all expire).

Anyway, this is just arguments. I think your code is quite nice, and certainly instructive. I just wanted to leave the note for others that if you don't know enough to make your own decision between "roll your own" and "use the thing that ships with Plone", you probably want to use the thing that ships with Plone.

Martin

Re: Daniel Nouri's Blog

Laurence Rowe — Sat, 12 Feb 2011 14:14:33 -0000

I'm also a big fan of custom caching logic for high traffic sites, bu for me it seems much better to put that logic into your cache's configuration rather than in custom python code. Varnish has a wonderfully expressive procedural configuration language which gets compiled down to fast C code. As it runs on every request you have much more flexibility than just setting cache headers in your application. (It can be useful to set an X-Portal-Type header to give you some more information to play with.) It also tends to be much more simple to test and deploy changes to Varnish configuration than it is to test and deploy changes to application code.

We can all agree that CacheFu was an abomination of monkey patches, but plone.app.caching is much better. It means you don't have to spend time configuring standard things like caching resource registry requests forever.

Re: Daniel Nouri's Blog

Daniel Nouri — Sat, 12 Feb 2011 13:11:07 -0000

> ... that plone.caching/plone.app.caching are the supported caching tools that ship with Plone 4.1 (and work in 4.0 as add-ons). A lot of time has gone into making sure these are robust and complete. For people as experienced as you, writing your own hooks may be preferable, but I think for most people, using a core Plone package (and a GUI) is going to be the safer option.

I'll take the argument that one should use plone.app.caching because it ships with Plone with a grain of salt. I remember CacheFu itself used to be shipped with some flavours of Plone. ;-)

Also, I wonder what the point is in going for a "safe option" when you need to deal with lots of traffic on a slow CMS. Safe isn't gonna cut it then. But sure, if you'd rather not think about caching, turning on plone.app.caching is better than nothing.

> There's no support for 304 responses in your code. In my experience, 304s often provide the best tradeoff between speed and staleness ...

Yes, my code doesn't do validation, it only does freshness. With Zope and Plone, doing a conditional GET is almost as slow as delivering the actual content.

In my example, every HTML page is cached for one hour and there's no validation. That's because validation often isn't very useful and it's hard to get right. However, Varnish is configured to always deliver fresh content for logged in users.

> Caching based on portal_type means you get the same policy for all views of that type (e.g. the public view, a folder listing and a page). That may not be optional, especially if logged-in users also need some degree of caching.

You're right. If you allow public visitors to access more than just one view, then you'll need to add that to the picture (to your if statements).

In my example, we only cache resources, never dynamic content for logged in users. (If you're dealing with a site with many logged in users, you'll need to overthink that strategy of course.)

Thanks for your comments.
Daniel