As many of you are aware, there is a SERIOUS browser war going on. All of the major players are actively seeking to carve out their share of the browser spectrum.  The never-ending nightmare for web developers has grown due to the fact that all of these companies have introduced completely separate (but similar) technologies to enhance the existing browser.  All of them (Adobe, Microsoft, & Google), expect the browser to become more than just the simple document viewer it once was; they anticipate it (along with many others) to become the standardized environment where all (or most) future applications will run.  What does that mean?  Well…it means that new applications will be developed and supported cross-platform by way of the browser.  Companies will distribute, service, & provide updates to their applications, entirely through the web.   This sounds great; however, browsers are simply not powerful enough to handle complex applications.  Furthermore, current web standards do not support communication with your local machine/file system.  So how will we make this leap forward in browser technology??  Well, by applying pressure to the consortium of web standards (w3c) through the use of browser plug-ins of course…

Adobe Air – a cross-platform runtime environment for building rich Internet applications using Adobe Flash, Adobe Flex, HTML, or Ajax, that can be deployed as a desktop application.

Google Gears – is an open source project that enables more powerful web applications, by adding new features to your web browser.

Microsoft Silverlight – is a programmable web browser plugin that enables features such as animation, vector graphics and audio-video playback that characterize rich internet applications.

Unfortunately, due to the nature of competition, it appears that we are moving farther and farther away from standardization.  Each of these plug-ins not only requires a unique download and installation, but also utilizes an entirely different set of code for the developers.   As you can imagine, there is timidity amongst web developers/designers when choosing which plug-in to use.  Implementing all 3 of these enhancements into a single website would certainly result in substantial development cost; quite frankly it would be impractical.  So most of us in the web world are left with a choice!

At the moment, in order to achieve widespread support, noog.com does not utilize any of the above mentioned plug-ins.  Because Silverlight and Adobe Air applications are developed with somewhat proprietary languages, the noog platform is most well suited to integrate Google Gears.  Why you ask?  Well, because of simplicity really.  Gears can be effortlessly integrated into an existing website by including the standard Gears javascript library.  Developers can interact with the Gears runtime using ordinary javascript (something our team knows fairly well!).  Although Gears is still in its infancy, it currently allows web developers to take advantage of a visitor’s local environment by utilizing a local database system (SQLite).  Gears also allows websites heavy in javascript (like noog!) to use what Google call “worker pools.”  In an effort to speed up javascript, worker pools (in essence) allow developers to “multi-thread” their costly javascript operations by sending them to the “background,” thereby curtailing costly javascript “hang-ups.”

With its open-source approach and integration with Google’s new browser Chrome (Chrome automatically comes with Google Gears), we believe that Google Gears will become the mostly widely utilized browser enhancement over the next few years.  With its ease of integration and alignment with our overall vision for the browser, you can expect to see the noog platform make use of the power of Gears in the coming months.

The past few weeks have been craaazzzzzy! Many of you may have noticed some random server outages (sooo sorry)….we have been frantically working to resolve some lighty issues that have been causing our entire system to crash….boooooo

But the good news is, I believe we have isolated the issue and resolved it (maybe??). Things have been running smoothly for the past 48 hours….we will have to wait and see my friends…

But most importantly, we have just reached 7000 users!!! Since our alpha 2.0 release, and subsequent nationwide exposure, noog has really started to gain traction on several schools across the country. We know we have a LONG LONG way to go but it is important that we celebrate along the way…..right?? So thank you…..to our first 7000 users. We know that things haven’t always been working the way they should, and features are still missing; however, don’t worry folks…..we are committed to making this platform an essential tool for students across the country. Our development team has grown again (2 people added this week!). Some beautiful applications are on their way

Opening hailing frequencies…

Hi, my name is abinoog and I manage the servers for noog.com when I am not busy playing starcraft with my college mates :).

I am here to give you guys a quick walk through of how we manage load balancing for noog. As you all know our second alpha release brings significant enhancements and features to the front end. However, what you probably do not know is that there have been significant modifications to the backend too. Our server architecture and the method in which we serve content has been radically altered to increase throughput, reliability and speed, so that at the end of the day you can walk away with a great user experience and visit us over and over again :).

In our first crude attempt at load balancing we simply forwarded you to one of our particular servers.  Up until a few weeks ago if you visited www.noog.com it would redirect you to www1 or www2;  you would then stay on this particular server (www1/2) forever.  All of your requests would go to the same server.

One of the reasons for doing this was because of the way we stored our session data.  At the time, our sessions were server specific.  What this means is that if you logged in on www1.noog.com and now go to www2.noog.com the you would have to login again on www2.

Large websites play a number of tricks to remove session stickiness such as storing sessions in the database so that they can transparently balance the load of your requests across multiple webservers. However, databases due to their inherent nature are often large monolithic beasts that offer considerable flexibility and power when it comes to data manipulation. Hence, at times its an overkill to use them for storing session based data that may need to be pulled regularly.

It is also important to understand that a call to the database for pulling data essentially traverses all the layers in the entire application serving stack. While this is not something which can always be avoided considering that the database is often the slowest link in the entire chain it reduces the speed at which we can serve requests to you. So as part of our application design we try and reduce the number of calls we make to the database as much as possible.

Enter memcache, gooze has already talked about the beauty of this and how the development team is using it so I won’t go into a lot of detail about it.

I will however touch a little bit on how it allows us to remove session stickiness. Essentially since sessions are stored in memcache when you login to noog it allows PHP to access them from anywhere no matter which one of our servers your session resides on.

This simple change opens up tremendous opportunities for enhancements at all levels of our application stack including the ability for us to perform TRUE load balancing where one request sent by your browser maybe served by one server while another request may be served by a different server. So basically you continuously hop around our servers as they serve you :).

This hopping can be controlled by a number of algorithms and in the coming months we will be fine tuning our hopping algorithm to ensure that load is evenly balanced on all our servers.

I hope you enjoyed this little piece and now have a better understanding of what goes on behind the scenes to bring you the noog.com experience that you have all come to love and enjoy

Where do I begin…speechless really….memcache has truly wowed me.   This “simple” daemon is all I think about nowadays.  Noog has officially taken a giant leap forward in our goal of supreme scalability due to the hard work of those at live journal.  What an awe-inspiring piece of grandeur.

So what is memcache?  Before I explicate, let me first warn those experts out there.  If you are “elite,” or are a maharishi in the ways of web development, then please pass over this cushy explanation…it will surely be beneath you.

Okay, so for the rest of us noobs (myself included), lets get down to the elegance that is memcache (pronounced “mem-cash“…or at least that is how I say it).  Traditional web applications are comprised of 3 layers:

1. The top layer is the front end user interface that you actually see… generally comprised of HTML, CSS, JavaScript, etc.
2. The next layer is the all-important “middle man” between the front-end and the back-end data storage system.  This layer is usually written in a common scripting/programming language (PHP in our case) that allows dynamic content/pages to be created on the fly.
3. The last layer is the backend data storage system (database) which holds all of the site records like user, product, and page information (noog uses a MySQL database).

So where does memcache fit in??  Well, the answer is between the 2nd and 3rd layers listed above.  What the people at live journal found was that frequent requests (queries) to a centrally located database (usually on a single server) produced a significant load on that server….why?…because generally when you “query” a database you are asking it to perform a “search” on a list of data to retrieve your results.  This “search” often times takes valuable time and critical processor cycles.  Aside from the high query cost, databases like MySQL are fairly difficult to scale to multiple machines.   Sooooooo…massive sites like facebook run into serious scalability issues when queries are so frequent.

Memcache to the rescue! The concept is simple, memcache is an added layer of data storage that is placed entirely in a server’s memory.  Additionally (and most importantly), memcache is built as a distributed (across multiple machines) memory storage system.  Memcache allows you to easily link up several servers (Facebook’s has 800+…last I heard) and utilize the combined memory across all of them in order to store any frequently accessed data across your site.

So instead of having to perform numerous database queries to a centralized source all the time, you first see if the data you are requesting is stored in memcache.  If the data is found, memcache will return it back much quicker than any database query because:

1. the data is entirely in memory (mysql supports this as well though)
2. a “pre-made” compilation of data that can be pulled up without any form of query/search because of memcache’s hash key based search system
3. the load is distributed across many machines.

***EXPERTS START READING HERE***

So how does noog use memcache?

Every user inside noog has several attributes (name, noog id, online status, school, thumbnail url, etc).  This data is frequently requested from our mysql database anytime a user’s information is displayed (people search, course forums, etc).  Aside from the basic member information you also have user-specific block lists and privacy settings that need to be checked every time a noog user’s information is displayed.  As you can imagine, this data is stored across several different tables inside our mysql database.  The query to retrieve all of this data is certainly not very cheap.  What memcache has allowed us to do is to store our own “noogUser” object.  This “noogUser” object stores all of the user’s data (including the blocklist, privacy settings, etc) inside one entity.  Now….in order to retrieve ALL of the necessary data for a particular user, you simply ask memcache to give you this object (no massive query anymore).   If memcache cannot find this object, then it will perform the expensive database query (one time) in order to populate the “noogUser” object and return it to you.  The next time someone requests this same object it will already exist in memcache and will be returned instantly.  Make sense?

Now you may be wondering…what happens if someone changes their user data….do the store objects in memcache expire?

If a user changes their personal data, first a database update will occur, and then a memcache update will occur.  That way you always have a “hard copy” of the users information inside the database in case memcache fails or the server must be rebooted.   In our case, we do not have an expiration placed on our noogUser object; however, memcache allows you to easily specify how long you want objects to exist in memory.

That’s all you use memcache for?

Not at all my friend….we are working on using memcache everywhere.  I told you I dream about it nightly didn’t I?  Another extremely valuable way we have utilized memcache is by storing various lists of users.  Throughout our website we frequently test to see if another user is your friend or classmate (coming soon:  roommate, sorority/fraternity sister/brother, etc).  If we had to query the database every time we wanted to know if someone was in one of your classes, or is your friend (as you can imagine) it would be quite costly.  To alleviate this expense, we have created a user list object that we also store in memcache.  This user list object contains an associative array that allows to quickly run a test to see if the user in question is in this list or not….so beauty.

We are currently using memcache to store all sorts of objects (will post about them later if you all are interested!), not to mention all of our session data.  Memcache will certainly be an integral part of our application going forward…

What do you think???

Avid users of noog may have recently noticed our newest addition to the noog environment, our auto-complete (or auto-suggest, auto-fill, etc) module (found on our people search, our professor search, and our friend search inside the profile application).  In case you have no idea what I am referring to, an auto-complete module is simply an input box that displays results to you as you type.  We know that this type of input box has become common place across the web world; however, I just wanted to share with you, the exquisiteness that makes up our “be all and end-all” auto-complete.

Modularized…

That’s right, the noog auto-complete is one centralized module that loaded in the parent document (window.top).  It can be utilized in any of our applications, including within our applications that are loaded in iframes!  But wait….how can we take code from the parent document and use it to load a module inside our iframe you ask…especially since the auto-complete uses relative positioning when displaying the results??  Well my friend, the answer is some serious trickery.  Let’s say you want to create a new auto-complete input box, and you want to place it within your iframe without actually including the javascript code inside your iframe.  Do this you simple call our autocomplete module directly from window.top…with one caveat…you must pass it a reference to you “document” and your “window.”  Example here:

var myNewAutocomplete = new window.top.autocomplete(document,window);

By passing a reference to the document and window properties, the module will be built inside your iframe using the code originally loaded in our parent noog document.  We certainly plan to use this same modularized approach to standardize various tools used throughout our platform…cool huh??

Customizable…

This auto-complete module is loaded with customizability.  Aside from basic color changes, this auto-complete module allows you to include images (stole that idea from Facebook, sorry!) and subtext.  Static results, size of results, and minimum/maximum width for the drop down can all be set.

It’s own search engine??

You heard right, this auto-complete module also has a built in search engine and data storage system that provides the ability to “pre-load” your data set results (instead of doing a standard server request every key stroke).  Once the data set is loaded into your module, it then creates an index that can be QUICKLY searched (obtaining fast search speeds took several days to solve).  Aside from the initial server request, no other requests will be done. Additionally, when the auto-complete module is in “normal” mode (server request based), it also utilizes this search engine to store previous results and to figure out future results if the last search returned less than the maximum result number (usually 10).

“be all and end all” worthy?….you tell me!

The summer of 2008 has been deemed the “make or break” period for noog.  It’s not just about the website and the users, but also the development team, our efficiency, and our ability to grow and work together.  While students are out of school, the noog team is using the summer to overhaul its operations.  Thus far, the feedback we have received is excellent.  We know….we have been a little negligent with the bloog for a while now, but don’t think for a second that nothing is going on!

It is now time to perform; you all can expect that we are working overtime.  The next version of noog (v. 2.0 alpha) is scheduled for delivery in early September.  Our goal is to get these new features and interface tweaks to students as they start school again.  We encourage everyone to play with the current functionality and give us feedback!

Our new taskbar has been finished and released to the live environment. We realized several months back that as more applications are added to the noog desktop, the ability to manage all of the open windows (especially while chatting) will become increasingly difficult. Previously, with multiple windows open, there was no easy way to access windows that are behind others. Instead of only displaying minimized windows on the bottom, we decided to create a “windows-like” taskbar that shows all of your open windows. You now have the ability to flip between windows with ease!

The elements in the taskbar also feature:

  -  right click menu that allows you to quickly minimize/maximize/close windows
  -  flashing green alert (when a chat message is received, 
      the taskbar element flashes to let you know)

Contrary to what you may think, this was a fairly substantial update to our entire windowing system! What do you think??

As you may know, the noog photo sharing system has been under development for the past few months.  Here are just some of the features of our upcoming photo sharing system:

- unlimited photo uploading
- ability to share photos with other websites & allow non-users to view your
  pictures (if enabled)
- picture comments, tags, and ratings

We hope to have the PhotoBox system added to the live environment in the next few weeks.  Take a look at the screenshots below for a sneak peek!  (click on them to view the larger version)

Get excited…

We have been working to solve various user disconnection issues over the past few months.   Obviously, you shouldn’t be disconnected if you have a stable internet connection.  With our last update (released last week), we made some fairly significant changes to the disconnection handling.  Our next release will feature an auto-reconnect feature that will attempt to re-connect to our system every 30 seconds (attempting 3 times).  

Please let us know if you are currently experiencing any issues!

A few of you have mentioned that the blinking cursor is missing at times for certain input boxes (particularly in the login window).   Apparantly this is a known issue with firefox (https://bugzilla.mozilla.org/show_bug.cgi?id=167801). 

From what I have gathered, the cursor may go missing when you have a text input overlapping a div that has the overflow set to auto.  The good news is that I have utilized a “workaround” that should help solve the issue for firefox users.  The “workaround” involves placing the input element inside a “wrapper” div that has the overflow property set to auto.  Seems to solve the issue (for now!).  The latest version of the noog platform (including this fix) will be released within a few days. 

Boo firefox…is that blasphemy?