MattL.com

RAD: Rapid Application Development

One of the things that drew me to the web was its immediate gratification model: make a change, refresh the browser, see the result, rinse, repeat. Over the years, I have come to love and hate ASP: its object model pales by comparison to PHP, and its support of OO design borders on pathetic. Yet COM affords it quite a bit of flexibility, and ADO is the cat's pajamas when it comes to data access. So, there I was back in 1999, humming along with my ASP when word of ASP+ first circulated...

ASP+ was the brainchild of a couple of MS ASP guys (and I can't find jack about them now) who sought to address ASP's biggest shortcoming -- it's a scripting language. They, as I recall, in two weeks made a JIT-compiling version of ASP. Oooh, ahhh... there goes ASP's biggest hindrance when stacked against JSP! There were hints of "server controls", too, that eliminated the tireless <%= %> mixed into my HTML. Then the Microsoft PR .NET Machine got wind of it, and, wham, bam, *flush* -- ASP.NET, a complete rewrite of Microsoft's web serving platform, was born.

ASP.NET suffers from Microsoft's typical affliction -- over-architecting. Now, making any sort of "smart" web page requires a Solution, a Project, a WebDAV-enabled server, a WebForm, a CodeBehind script, and a bunch of little poo directories spread about just for fun. And you STILL can't debug cross-server without having the stars aligned just so. Yes, the .NET Framework rocks all, but I just.want.to.make.a.damn.page.

It's funny, but even four years after its release (and 5 since a PDC beta), I can still sense the bloat and awkwardness of an ASP.NET web site. The back button gets jagged up nine out of ten times, form validations appear as oddly placed labels, and some developer left ViewState enabled so I get a 1MB page refresh for my troubles. Yechh.

So what are the alternatives?

PHP? I've done my fair share of hacking in PHP, and it's no picnic. "Code soup" hardly begins to describe the Perl-esque object model (which makes sense, BUT). Where the .NET Framework is a hierarchical model, PHP is the biggest, flattest tree you've ever seen. 7,000,000 top-level objects and counting. Anyway, it's an OK language, and if you miss pointer notation for some ungodly reason, it's an option because someone has coded some hackaround for you to consume just about every type of object out there.

JSP is Java, which in my mind means codebreaking compiler changes with every revision. Thanks, but no thanks. We danced that way once, Sun, and I won't have my heart broken again.

Python is cool. But that's all it is (to me): cool. As with all of the grassroots languages, there's the sense that it's due for a rewrite any day and something you wrote last week will break. I don't like that.

Perl. Heh, c'mon. Perl is soooo old school. Speaking of which, there's probably a Perl::OldSchool module in CPAN. <eyeroll> It's the same people who want CGI to live forever.

Ruby. Man, what is up with Ruby on Rails? I looked at this language back when it was fledgling and someone somewhere said "hey, check this out". I wrote a couple of pages in it and never looked back. How on Earth these people are calling it godsent is beyond me. If you're a big fan, feel free to let me know why and what language you come from, because I'd love to understand the attraction.

Matt's hackaround

ASP.NET on ASP. Ha! Okay, it's not as catchy as "Ruby on Rails" (which sounds like a band), but it's descriptive and in the same vein. Most of my front-end code is written in ASP (using ASP's Cookies), while the back-end code is accessed through ASP.NET Web Services. In this way, we can test some new whiz-bang on ASP and, if it looks like it's going to stick, work the code into the back-end services with a built-in round of refactoring.

PS

I use Movable Type for this site, which is written in Perl, because it's one of the supported blogging platforms for Windows Live Writer. Go figure.

I recently attended the Connect conference hosted by Inman News in San Francisco.  It's a once-a-year event for the Real Estate technologists to get together and talk about the geeky stuff that the rest of the Real Estate community doesn't understand.  Notably present were the major web-based mapping providers -- Google Maps, Yahoo Maps, MapQuest, and, of course, Microsoft (en force).

My two missions for the trip were:

1. Scope the announcements from the Big competitors, and
2. Build a strategy for adding mapping to our application in the next year

Mind you, I went through the same exercise last year, but the cost:benefit ratio didn't create a compelling argument for adding any sort of data-integrated maps into our service.  Now, however, the Big guys are offering integrated mapping, and the providers (especially Google) are offering a much richer object model than they did previously.

I most wanted to hear from Microsoft: I'm a Microsoft platform coder, I'm six-ways certified, and I actually enjoy writing against the .NET Framework.  I wanted the Microsoft guys to tell me there was a reason not to use Google.

I emailed Chandu Thota the night before the conference to check on his availability and whether he was up for a chat.  He was, and we did, along with Virtual Earth Program Manager Steve Lombardi.  I explained to them how the Real Estate information ecosystem works, and offered what we, the Real Estate vertical, need from a mapping solution.

There was one point that Chandu kept hammering on -- JSON:
"It's lightweight!"
"It's object-oriented!"
"It's cross-browser!"

It also breaks this Holy Commandment of Programming that has been pushed by every major platform vendor for the past five years (especially MS):

THOU SHALT NOT MIX INTERFACE DESIGN WITH CODE.

I learned XSL for a reason: It's the Right Thing to Do™.

So why, praytell, would I make this two-step-back advance to JSON?  Bandwidth?  C'mon, if you're sending 500K in data during a single action, then you have an application design problem.

I went on to prove this point by mocking up Google vs. Virtual Earth maps, with a single dataset through various methods of access (static JavaScript, JSON, XML/XSL, and JSON-to-XML/XSL), and y'know what?  The difference in over-the-wire data transfer and presentation methods was far less significant than the platform's rendering method.

Give back my Model-View-Controller holy trinity and stop trying to re-invent the wheel.

I have since coded up a JSON-based AutoSuggest for our service, and I remain utterly unconvinced:  oDiv.appendChild() should not be a design technique.

Ah, the wicked mistress that is JavaScript.  It has been years since I really got into JS; sure, I've made my share inline popup dialogs, AJAXified pages here and there, but nothing really in-depth since the bad ol' days of IE 5.5 and the DOM nightmares that would wake me in a cold sweat.  I really didn't miss that aspect of JavaScript; certainly no moreso than a POW misses the war.

Well, it's a new world nowadays (old-timer talk):  The DOM is almost universal and JavaScript has grown up... way up.  Actually, JS hasn't grown up so much as the platforms (mostly) comply with the ECMA ratification from all those years ago.

Don't get me wrong -- back in the day I loved JavaScript as much as I loved my first new car.  It was like Java and C++ without all the hassle, and you could make the web do tricks at a time when it was 99% static (god, do I miss those days).  BUT, the aforementioned pain of cross-browser support meant hours and hours and hours of testing, tweaking, implementing tiny code "shims" to get stuff to work right ... it almost wasn't worth it in the end.  The original menuing system at Pergo.com (dynamic JavaScript with HTML drop-downs loaded by Vignette that was 6-language friendly) took me weeks to get working properly.  In the days before JSON, it was quite the technological feat to have all of those disparate points of navigation play nicely as a huge multi-dimensional array.  It worked something like this:

pergo_menu[0] = new Object();
pergo_menu[0].text = "Interior Design"
pergo_menu[0].url = "../b/b1_interiordesign.html"
pergo_menu[0].items = new Array();

pergo_menu[0].items[0] = new Object();
pergo_menu[0].items[0].text = "Inspiration"
pergo_menu[0].items[0].url = "../b/b2_inspiregallery.html"

pergo_menu[0].items[1] = new Object();
pergo_menu[0].items[1].text = "Q&amp;A designer"
pergo_menu[0].items[1].url = "../b/b7_talktodesigner.html"

That was all the backend guys had to do to update the nav.  Not bad, even by today's standards.

So last week I was back "home" in New Hampshire for a long weekend of R&R and some great quality time with my three younger brothers.  One friend of my brothers, Lokesh Dhakar, had long had web dreams, and I'd heard through the grapevine (brother #2) that he'd gotten a job offer from Google.

Me: "Hold the phone... Google?  Lokesh?  Google?"
Brother: "Yeah, he invented some kinda new popup thing that everyone in the world wants."
Me: "Uh, okay."  (brother is not web-savvy)

In the night of formal brotherly carousing, friends from near and far gathered at a local tavern, including Lokesh.

Me: "Hey, Lokesh, what's up?" (we've always been friendly, although I used to be the cool web guy... now the shoe's on the other foot)
Lokesh: "Hey, Matt!  Good to see you."
Me: "What's this about you inventing popups?"
Lokesh: "Well, that's not really it.  I made this thing that pops up on pages and makes it really easy for anyone to implement.  I got calls from Yahoo, Google, Microsoft, and a bunch of others.  That's how I got my current job."
Me: "Uhh, popups?"
Lokesh: "Well, it just made them easy.  Check out my website and Lightbox and you'll see."

To me, Easy Popups != Google job.  There had to be something more.

Indeed there was.

Lightbox leverages the prototype library, script.aculo.us, and a whole lotta DOMmin'.  The result is a really, really, mindlessly easy way to add a pretty "see larger image" link on a web page.

About two hours later, I was incorporating the concept into our web service's dev system.  I've long done "pop-in" dialogs with IFRAMEs and a PNG shadow, but the effect of a pseudo-modal, smooth interactive element is a thousand times better.

So now I've been JavaScripting for two straight days, loving and hating it all over again.  The remaining burning question:

WHY ISN'T THERE A REAL JAVASCRIPT EDITOR YET?

Seriously.  The language is there, the platforms are there, but for the love of king and country I still can't debug it without silly elements named "tracediv" and a whole bunch of nodeValue calls.

Debugging in IE is still a joke, even with the developer toolbar (please stop overriding CTRL+R, thank you).  That being said, I still haven't loaded the Venkman JavaScript Debugger...  That's for tomorrow.

In this post at the Mini-Microsoft blog, Mini makes the argument that there are too few "qualified" applicants for the Microsoft programmer ranks, citing that they should be educated in "C/C++, Win32, COM, ATL, XML, DHTML, AJAX, .NET, debugging, performance, Watson analysis, design patterns, security, using our best internal tools and resources and so on."  And instantly I understand why so much of VS.NET's auto-generated code is terrible, and why their Atlas library requires a developer "think like Microsoft" (answer: because no one else does).

Further:

[...] more and more candidates who can lay down the smack with Java and script can't manipulate memory and discuss deep operating system constructs just-in-time at all. I need you to be able to write a GC, not be in an unhealthy co-dependent relationship with one.

It's that sort of painfully dated thinking that maintains Google's ownership of web innovation while Microsoft struggles to be noteworthy.  A talented, creative scripter has absolutely no domain over deep memory management and should never have to think about it: That's why you have talented, deep memory guys in the first place.  What's the purpose?  Advancement?  I'd start looking for other apocalyptic signs if any of the AJAX wizards I know hint at working on some nice kernel code; I would no sooner expect a podiatrist to be studying neurosurgery.  While collectively still "programmers", many of us have highly diverged specialties sharing little in common but tools of the trade.

Don't get me wrong -- being familiar with the clockworks required to make your code work is a Good Thing™ and often leads to higher quality code, but it's no substitute for creative thinking and one can argue that the focus gained by an "unencumbered" mind is equally valuable.  As a master generalist myself, I tend to see my greatest shortcomings while overthinking problems and overarchitecting solutions -- a problem that doesn't plague those with shallow skills.

Perhaps Mini's viewpoint is a one-man reflection of the great cancer that keeps Microsoft from being great, though it's terribly ironic that it's coming from the guy so passionately trying to instigate change.

Out of last week's (excellent) RETS conference came the ultimate software question:

Do we rewrite RETS in a new incarnation, or refactor RETS 1.7 to eliminate the (numerous) performance and implementation problems?

Along these lines, Chris McKeever forwarded a classic Joel Spolsky post titled "Things You Should Never Do, Part I" (oddly, there is no Part II) on the dubious merit of scrapping & rewriting any piece of software.

I mostly agree with Joel; however, I have some apprehension because there are examples of (very) successful rewrites: MacOS to Jaguar, Adobe Photoshop, HTML, eBay, MySpace, and so on ... and one purpose in those rewrites resonates with RETS 2.0: a fundamental change in the underlying architecture.

RETS 1.0 specified everything -- implementation, transport, protocol, and data standards. About all it didn't do was include a specialized implementation of TCP.

RETS 2.0 is light fare by comparison -- a data and interface description standard. Most of the plumbing (transport & protocol) that made 1.0 complicated is supplanted by existing Web Services standards: HTTP, XML, authentication, and the atomic transaction have all been written by other, smarter (or at least more directly focused) people.

Should RETS 1.x be refactored to current standards?

Pros:

• RETS 1.5 & 1.7 are implemented in a majority of the software in the market.
• RETS 1.7 can be "wrapped" in Web Services.
• Significant work has been done by the CRT (and a few third party ISVs) to make RETS more accessible (libRETS, ezRETS, Variman, Viele, and so on).

Cons:

• RETS 1.5 & 1.7 are implemented haphazardly, and almost each implementation has some sort of workaround to disambiguate part of the specification.
• It is very, very fat with overhead (even if not sent over the transport because of the compact format, it still requires implementation & maintenance).
• No supporting documentation is provided beyond the single, 130-page specification which is rife with wishy-washy "MAY"s.
• Antiquated (to say the least) DMQL.
• People have been trying to "fix" RETS since its inception, and it just ain't happenin'.  Why kid ourselves with the idea that we can resolve a decade's worth of issues inside of a year?

Yes, RETS 1.5 & 1.7 can be patched to resolve some of the ambiguity problems (at least the low-hanging fruit).  However, how much effort (and vendor buy-in) will go into maintaining three parallel versions of RETS?  Why would a vendor who is stuck on 1.5 choose to implement 1.51 instead of 1.7?  Why are we giving them an "out" to not update their code?

Yes, RETS 1.7 could be wrapped into a Web Services framework and appear modern at the surface, but that draws all the worst "lipstick on a pig" analogies you've ever heard (or, if you haven't, "you can put lipstick on a pig, but it's still a pig").  How many hoops will we have to jump through to get it working correctly?  Remember that such a solution creates at least three layers of obfuscation to the actual data: three layers of code, potentially from three different vendors, all with bugs or, worse yet, special fixes to the other vendors' bugs (increasing fragility).

Call me old school, but I want one, monolithic, no apologies, no exceptions standard.  Make the service contract very explicit, and do not grant compliance certification to anyone who fails for any reason.  Write some white papers, get someone to write a book, and have a half-dozen professional data architects go over the spec to make sure we're doing the right thing.

We need to make people (programmers and consumers) want to use it and ask for it by name.

Michael Wurzer posted (over at the FBS Blog) his commentary on the ever-roiling hype around map-based searching in MLSes and for real estate consumers, borne out of Real Tech LLC's implementation of polygon-based mapping for John L Scott.

Whenever I ask our client base what features they'd like to see added to our service, map-based searching (MBS, for short) inevitably takes one of the top three positions.  However, when pushed with the 5 Whys, virtually no one can get past "because xyz does it!"

I have closely followed the various online mapping options since they became financially accessible and widely accepted (roughly 3 years ago). However, based on numerous seminars and conversations with end customers -- as well as top people at the mapping vendors -- the reality is that mapping hasn't either A) matured sufficiently for, or, B) proven its value in, our market.

As I've pointed out on in several RE forums, I'm skeptical at best of the merit of broad, location-based searching. I personally believe the value is in context-based geography, such as "maximum distance to x school, y office building, and z office building". The problem is then the processing power (and licensed software) required to provide those results in a sub-second period... not an easy trick when you have to scale to thousands of searches per hour.

At last year's Connect SF, I had the opportunity to discuss at length the issues surrounding MBS with Microsoft's Live (nee Virtual Earth) Maps team.  Out of those conversations, Microsoft added off-map drive-time calculation for points (which Google also implemented mere days later).  Unfortunately, the reality of having to load several hundred candidate points and perform the calculations against their servers still proved too latent an exercise to be useful.

The average real estate consumer -- the should-be target of these features -- simply does not rank the featureset nearly as relevant as price, bed/bath count, and square footage... especially when beginning their search, which is how most of these tools approach the visitor.  In fact, I think most consumers will sacrifice location in return for a house that otherwise meets every need.

This year, we will add interactive mapping to the Favorites feature of our service, only because it will include multi-point route optimization (for planning drive-by tours and showings).  To me, this is the extent that the technology has provided value to the consumer.

I would much rather let the big corporate vendors spend millions figuring out what works, then implement those features for a few thousand dollars in a quarter of the time.

Nearly every developer is a diamond in the rough in some way; rarely is a new employee a perfect fit for the job, even when their resume says so.  I think this happens for a few reasons:

1. Resumes (and sometimes even blaring overconfidence) exaggerate most candidates' skills.  Most often, it's because they knew how to pull every handle and turn every knob at their last job, so they must be an expert... when, in fact, they were just experts at their last job.
2. Your job opening is almost never the same as their last job.  Unless you're hiring laterally from another department or absconding an employee from a direct competitor (who has the same platform and build process as you), your job will have unique requirements and being productive in that job will require implicit knowledge of how your company or department operates.
3. "Perfect" is a subjective mix of wisdom, efficiency, insight, overall productivity, communication, and socialization.  It's impossible to be Perfect on Day One, if not Year One.

Recognizing this, I try to keep an open mind when hiring new staff (which I've been doing tirelessly for the past two months).  There are really only a few traits I key-in on:

1. Appropriate level of knowledge for the platform.  I've been hiring for a .NET developer with a modest SQL background.  I don't particularly care if their experience is in C# or VB[.NET], I just care that they've had solid experience with the framework itself, which is an immense geography of 2500+ classes and tens of thousands of functions, so in particular the parts that we use most (data and IO).  I check mostly for awareness and a keen sense of what's-what (like the difference between Inherits and Implements).
2. Good programming.  Beyond linecoding, being a good programmer means that you know how to back up, consider the bigger picture, and write what needs to be written (and nothing more).  God bless those uber-efficient linecoders who can churn spec into code for months on end, but you need three architects to keep them on track (which we don't have).  This one is harder to sniff out, but not impossible.
3. Raison d'être.  Some developers only write software to cash a paycheck. Ironically, most of them don't seem to realize that they're being outmoded by newer, faster, smarter developers who are eager for their paycheck.  Continuous improvement isn't just a corporate thing, it's a career thing.  I take it as a sign that anyone who hasn't flexed themselves in the past 3 years thinks they know all they need to know, and I don't hire those people.

All of this preamble was a build-up for my soapbox on Professional Development™.  It is far too easy for developers to stagnate, and they usually don't know they're a dinosaur until they're being "let go" in favor of newer/younger blood that has three times the depth in current technology and can't find a job.  Likewise, I'm there are many companies looking to expand their Lotus Notes & CORBA integration project from 1997 and want to see more "web 2.0" in it.  The reality is that those companies are mired in the 10-year-old mentality because they didn't help (or push) their developers into staying current.

At MLSPIN, I've been instituting the best policies I've ever learned working for various software companies, large and small, or heard of through online sources:

1. Conducive environment.  This is the hardest, since it takes a lot of money and there's only so much you can do with the facility.  My developers work in a bullpen (less than ideal), but have plenty of space and new ergonomic chairs.
2. Productive equipment.  Each developer has a real HP workstation, with a quad-Xeon processor (room for a second), 4GB of RAM, 15K rpm SAS drives in RAID 5, and an ATi FireGL video card (because they're better at 2D than nVidia).  Each also has two 20" flat-panel monitors and their choice of keyboard & mouse.  We run Visual Studio 2008, and each developer has a copy of Photoshop Elements and SnagIt.  I also have a 5-seat license of MSDN, so we can tinker when we're planning new stuff.
3. Progressive training.  In addition to the modern cachet of books, we provide $2500 in tuition/certification reimbursement, pay the certification exam fee for any passing score, and offer merit bonuses for accomplishing their certification goals.  I'm also committed to having two "superstar" training weeks each year.

The first training session this year was held in January and featured the inimitable Bill Vaughn.  Aside from being one of the founding fathers of ODBC and grandfather to ADO, Bill has the most comprehensive knowledge on programming against databases in Microsoft technology that I've ever even heard of.  Buy his books, they make you smarter.  In a week's time, Bill was able to convey as many nuances to the technology as I've learned in a decade of hands-on work, and I still learned new tricks.  He's very personable, doesn't believe in stupid questions, and knows how to mentor.  We liked him so much, we've now got him on retainer.

I still haven't decided who to have out for the August-ish session, so if you have any suggestions, drop me a line.

So, that's my pitch.  If you are a professional developer, do whatever you need to to stay current, and if your company doesn't support that, leave.  If you're a company who doesn't believe in training your resources, send me your developers' resumes.

-Matt

Today, it was formally(?) decided that RETS 1.7[.1] is about the best that can be achieved before the NAR's June 2009 deadline.  For the sake of edification, I'm including it here:

The integrity of data is a foundation to the orderly Real Estate market. The Real Estate Transaction Standards (RETS) provides a vendor neutral, secure approach to exchanging listing information between the broker and the MLS. In order to ensure that the goal of maintaining an orderly marketplace is maintained, and to further establish REALTOR® information as the trusted data source, MLS organizations owned and operated by associations of REALTOR® will comply with the RETS standards by June 2009, and keep current with the standard's new versions by implementing new releases of RETS within one year from ratification.

Now I'm going to do the posh thing and quote myself:

There is absolutely no chance RETS 2.0 will be done in fourteen months.

-- Matt Lavallee, RETS Assembly Meeting, Chicago, IL, August 9th, 2007

All I got then was hand-waving and chuckles...  Today, I may as well call myself The Nostradamus of RETS (although no one else will).

The ".1" incremental build is to correct some minor errata in the documentation of the specification.  This is truly pathetic:  RETS 2.0 was started in 2004.  How can anything be this slow?

For the sake of argument, let's look at the timelines of some other widely-known specs:

1. HTML
1. Version 1: 1993.
2. Version 2: 1995.
3. Version 3.2: 1997.
4. Version 4.0: 1999.  Note: Considered "good enough" by most of universe.
5. XHTML 1.0: 2002.
6. [X]HTML 5.0: 2008.
2. SQL:
1. ANSI v1: 1986.
2. FIPS 127-1: 1989.
3. ANSI SQL2: 1992.
4. ISO SQL3: 1996.
5. SQL1999: 1999.
6. SQL2003: 2003.
7. SQL2006: 2006.
3. Wi-Fi (IEEE 802.11)
1. 802.11: 1997.
2. 802.11a: 1999.
3. 802.11b: 1999.
4. 802.11g: 2003.
5. 802.16 (WiMAX): 2005.

I understand the issues of timeline management and pragmatism, but come on... this isn't rocket surgery.

Here's how I understand our current timeline:

• RETS 1.7.1 - Expected standards vote December 2008, adoption by June 2009
• RETS 1.8 - Expected standards vote August 2009 (?)
• RETS 1.9 - Expected standards vote... December 2010?
• RETS 2.0 - Prediction: 2015+

This makes no sense.