The Road to 1.0

Started by Gruffen, April 29, 2010, 05:10:07 PM

Previous topic - Next topic

Gruffen

The Road to 1.0

The road to 1.0 was a journey - and for some rather an emotional one. What follows is a combination of my recollection of how it went, plus my own personal views on things. This is not, and should not be considered as, representative of the entire team, even of the entire dev team.



Let me start with a graph showing the size of code development; it actually highlights the periods of development fairly well, I think.


First half of January (1/1/2010 - 15/1/2010)

This was one of the fastest phases of code production, just getting some of the very most basic code into place; the ticket template, the front page template, that kind of thing. I think we were all on something of a high actually, excitement and so on.

Some of the fundamental design decisions happened here and we just ran with them, not that any of them actually hurt us at all anyway.

During this part we also got the site founded, so busy time for everyone.

Development at this point:
* general ticket display
* ticket resolution
* action log

Second half of January (15/1/2010 - 31/1/2010)

I see a gap of almost a week in the commit log, and I know exactly why - the split of the SMF team occurred at this point, and I remember there was a debate over whether SD should continue building for SMF, or build standalone. You can tell what we decided to do, though, and I'm glad we did what we did - carry on for SMF only at this point.

Development obviously slower at this point because of this, but still some serious development going on:
* more on the action log
* foundation of posting done
* first stage of recycle bin
* view of resolved posts
* ticket to topic first added

It was at this point I also wrote the first version of our testing script, whose stats are not counted in the above.

So, really, more of the same, just with a slightly bitter feeling in the pit of the stomach.

First half of February (1/2/2010 - 15/2/2010)

This is where it started to get interesting. There wasn't a great deal of commits at this point where I was still figuring out how things like SMF's WYSIWYG editor component worked, though the very first 'new ticket' post happened in this phase.

The general feeling here was one of not quite being on the home straight, and a lot of separate elements that didn't quite all fit together properly.

To be honest, I had personal stuff going on; just before the 6 day gap in the logs, I lost my dad, and while it had sunk in there, it didn't really begin to hit me until early February, as preparations for the funeral were underway, arguing with the coroner about releasing the body and a bunch of other crap, so I didn't do so much.

At this point, we're up to:
* first draft of topic to ticket
* foundation of ticket posting
* first draft of posting UI
* some of ticket editing implemented

Second half of February (15/2/2010 - 28/2/2010)

This is where it went nuts. We'd said internally that we would go to beta on 1st March being the first release to the QA/beta testers and there was so much to do. You can see from the above exactly how much we did actually do, though, it was enormous.

In some ways this was my way of dealing with my personal crap, by shutting it out of my mind, and it helped me to just concentrate on coding, and the result of which was that I spent a renewal of energy on SD; ramping up from 8+ hours a day to 14-15 hours a day some days.

It was here that the strands were being pulled together, all the pieces of the jigsaw beginning to make sense at last.

Things added:
* attachments
* quick/'go advanced' reply
* quoting of tickets (which required ticket=123.msg123#msg123 support)
* ticket and reply editing
* track IP integration
* post previews

First half of March (1/3/2010 - 15/3/2010) - aka 1.0 beta phase

So after the mad mad rush through the end of February, up came 1.0 beta release.

This is the point where I would like to apologise to the beta testers, who got a load of crap from me that day. As you can imagine, we'd spent 2 months building it and all the feedback was negative.

Now, don't get me wrong, 1.0 beta was buggy in a lot of ways, and we knew that it was rough in a lot of ways. But I was expecting a balance of good/bad comments, good for the things people liked, bad for the things that didn't work so well, or were broken, and since I got no good comments, I was seriously wondering if everything we'd done over the previous two months was messed up.

And so being my emotionally charged sense I made a few less than nice comments, and I want to apologise to those who received them.

Virtually every commit at this point is either documentation or, more commonly, bug fixes - the vast majority of the changes here other than bug fixes were documenting the functions, in Doxygen format before finally converting to phpDocumentor format that we still use now (I have a set of custom templates to build it into the HTML you see on http://www.simpledesk.net/development/api/ though)

Writing the docs helped me get my groove back for the bug fixes though, and in short order we were able to push out RC1.

Mid March (15/3/2010 - 22/3/2010) - aka 1.0 RC1

I don't know about anyone else but I was pretty excited about RC1. There were a ton of bug fixes from the beta to RC1 as well as overhauling the main ticket listings to be sortable, which I thought was pretty worthwhile. (In hindsight, it's a piece of code that made an experienced developer exclaim 'yikes' when they saw it, which I'm not sure what to make of :P)

Even as I was producing the files for RC1, I had said to Nas in IRC that I thought there was something we forgot - and it turned out to be the case.

A day or so after, a fairly serious bug was brought to attention with respect to the recycling system for deletion of posts where things were getting out of order and unable to be restored. I'll admit, I was disappointed that 1.0 RC1 had something majorly wrong with it, since we were hoping to bound to final.

At that point I figured whatever we did would necessitate an RC2 to go because of the size and impact of the bug, so instead of just fixing any other bugs for 1.0 final, we sat back and looked at what we could do.

As it happened, the result turned out better, I think. The original implementation wasn't that far from SMF's methodology; deleting a reply gave you a new ticket with the deleted reply in, though differently to SMF we ensured that further deleted replies were kept together.

What we forgot, though, was that unlike SMF we don't have just a list of posts making a ticket; we have a single discrete post at the top that is the ticket - then everything else is a reply to it, so ticket should be handled differently, and brainstorming around it, we came to the solution we have now, that deleted replies are held in the ticket but masked. The result actually works better in practice.

I think I ended up just rewriting the system in a single day just to get it done, though it wasn't quite 100% - there was a bug we found after final in it, but it held up mostly to the changes.

Again, most of the changes were fixes, but we did add a few refinements - the ability to view a block only on its own, and collapse everything else was added here as well as the Go To Ticket dialogue.

End March (22/3/2010 - 1/4/2010) - aka 1.0 RC2 and the hopeful first 'final'

With this I actively felt like we were moving somewhere, and I could tell others felt the same way about it; someone commented that they "felt more able to trust it" with their data, which reassured me as to how it was going.

There were a few remaining silly bugs, which were fixed in short order, but there was one around attachments - if you upload an attachment that causes an error, the form was wiped out. Now, I was certain this was an SMF bug (and it does still affect SMF, as far as I know) and I spent a lot of time playing around with things, including finding a workaround or two.

In fact, I was so certain I'd fixed it, I went ahead on the 1st with preparing the 1.0 'final' package, except that unfortunately it didn't quite work out: the fix wasn't as thorough as I thought.

Half way through preparing the package, I found it wasn't as thorough, and with a lot of fury on my part for assuming it was fixed when it wasn't I shelved publication of the mod on the day we wanted to publish it on - 1st April.

Start of April (1/4/2010 - 5/4/2010) - actual 1.0 final

So, humbled by what had happened, I slunk back and tried to figure out a way to deal with what we had to deal with - and what occurred to me was that to a degree, in addition to fixing some of the issues, we could actually use the opportunity to really do something special.

So, browsing around the web I bumped into an example of the technique you see in the final code: using Javascript to manipulate the page, to do something quite different, so there's actually some checking of uploads *before* you upload them. The upshot of this is the bulk of error conditions are captured; unfortunately we can't check file size before upload. But it solved the issues we had, and looks and feels nicer to use than the current SMF code.

And thus we felt ready to actually go with 1.0 final, so we did on 5th April.

Then to now (5/4/2010 - 28/4/2010) - the road to 1.1

Well, we haven't been resting on our laurels either. Some bugs were found and squished after 1.0 final, for which a 1.0.1 package will be produced in the near future but there's nothing huge or calamitous in there.

Since then, though, a lot of stuff has been added for 1.1, with hundreds of lines of code added since 1.0 already for the new features already implemented (so far, ticket relationships, some of split/merge, some of custom tickets) and we're still going, if anything with more enthusiasm than at previous points.

And in case you're wondering how that graph looks based on today's figures, including the changes since 1.0 final...


For the geeks out there, here's the figures the graphs are made from:



Just to note, quickly, that I prepared all the images and table yesterday, and I did some more work last night so the figures are already out of date ;)

Yagiz

Wow, that's really amazing. Great work Devs :)

MultiformeIngegno

Man this post is fantastic!!!! :o 8)
Thank you!! :D
RockCiclopedia (wiki - forum - tv)
Tutta la storia del rock, scritta da voi ...

...iscriviti al feed di RockCiclopedia dedicato alle ultime news del mondo della musica!

flapjack

some people might finally realize, that it's not a piece of cake to bake such a nice piece of software. good work :)

[FailSafe]

Wow, SD development was more intense than I thought. So much going on the back-end that it can barely be conceived! You all make it look so easy. ;)

Great read, Arantor. And very fascinating to say the least. I'm sorry about what happened to your father though. My heart goes out to you and your family.

As for SD itself, the dedication that all of the team members put in is just a reflection of how great the product they come out with is.

Best of luck for the road ahead! :)

Gruffen

It was interesting to step back from the front line and look back over the history, actually. Though I knew we did most of the "big" development in February, I didn't realise that it was mostly the end of February that we splurged and just made it all happen.

One thing I will say is that we all knew up front how much work would be in something like this, and ultimately it's a no pain/no gain scenario. It took a lot of hard work to get to here, and I do think some people forget that, it's not just a case of throwing code together - if that were all it took, we'd be into 1.2 or 1.3 by now. It's also a case of testing, stepping back, tweaking and so on.

That's actually one thing about how things are right now: the beta testers can get a daily build of SD, which means they are able to evaluate new features the same day we implement them (the package is built at midnight, local time to me), so what happens is code gets written, by us throughout the day, and by the time I get up the next day it's been poked and prodded already, highlighting both bugs and potential improvements, and that's an awesome process; instead of building stuff and just having it 'This is how it is', we build it how we think initially it should be, knowing that it is likely a little rough in places, then it's shaped and smoothed off based on real user experience.


The one observation I didn't make above is that it's actually easier in some ways and harder in others to write a mod on this scale. The minute you step outside of SMF's files and are working on your own, there is something of a weight lifting going on in the back of your mind. Every edit to SMF core files is potentially something a user has to do by hand, and thus every edit is one you want to avoid if you can help it because it's one more barrier to support. Where you're using your own files, this is a non-issue.

On the flip side, when you're in your own files, you actually have to do more work because you're not leveraging SMF so much; we're using the fatal error that SMF does, and where it makes sense to integrate into SMF's code we do that, but the vast majority of the time you're interacting with SD it's through not only our custom code, but our custom templates too. That actually makes up a surprisingly large percentage of the PHP.

In case anyone was wondering what's happened since 1.0 final in those 2,000 lines of PHP - well 500 of those lines are comments on the code, which isn't a bad thing, but of the 1500 lines extra, there's bug fixes plus:

  • ticket relationships
  • splitting tickets (with some of the merge code done too)
  • some general revisiting of how a few things were done in Standalone mode, meaning no edits to index.template.php any more, and generally improving the install process
  • SSI functions for SimpleDesk
  • overhaul of the admin area, making items disabled when they aren't appropriate (e.g. hiding the list of possible bbcodes if bbcode is disabled)
  • a variety of smaller tweaks:
    • making IP visibility based on a permission rather than just visible to staff
    • icons next to attachments to denote what they are
    • a tickbox to close a ticket from the reply function

MultiformeIngegno

When users will begin to post like in the Aeva Media topic on SM.org (doesn't work, man it's all sh*t, help me it's not working at all, etc...) I think it's a good idea to give them a link to this topic!! :P
RockCiclopedia (wiki - forum - tv)
Tutta la storia del rock, scritta da voi ...

...iscriviti al feed di RockCiclopedia dedicato alle ultime news del mondo della musica!

Gruffen

In case you're wondering, Aeva Media 1.1b, counted using the same script I used to get these numbers:

Array
(
    [files] => Array
        (
            [images] => 107
            [php] => 45
            [xml] => 4
            [txt] => 7
            [css] => 5
            [js] => 5
            [other] => 6
        )

    [php] => Array
        (
            [blank] => 5225
            [actualcode] => 37423
            [comments] => 2538
        )

    [css] => Array
        (
            [blank] => 17
            [actualcode] => 918
            [comments] => 73
        )

    [js] => Array
        (
            [blank] => 65
            [actualcode] => 447
            [comments] => 32
        )

    [xml] => Array
        (
            [blank] => 64
            [actualcode] => 673
            [comments] => 47
        )

    [total] => Array
        (
            [blank] => 5371
            [actualcode] => 39461
            [comments] => 2690
        )

)


Yes - Aeva Media is over 3x larger than SimpleDesk based on line counts (not that that's everything, of course)

I think the Javascript count is off, though; IIRC the files are minified (ours aren't)

MultiformeIngegno

#8
I was meaning: when users will begin to say "SD doesn't work, nothing works for me, etc" with this graph they can realize all the work behind this project!! ;)
RockCiclopedia (wiki - forum - tv)
Tutta la storia del rock, scritta da voi ...

...iscriviti al feed di RockCiclopedia dedicato alle ultime news del mondo della musica!

Gruffen

Oh, I know exactly what you mean :)

I was just pointing out how much bigger AeMe is and why someone needed to reiterate this fact to the complainers there ;)

Mostly I ran the script on AeMe 1.1b out of curiosity.

Gruffen

I just wondered what the stats looked like since that snapshot, so I just pulled the figures from my dev build, which is about 40 lines of PHP more than the current SVN copy, while I'm still writing the feature.

And that amounts to:

PHP: 11,475 lines of actual code, 3,166 lines of comments
CSS: 533 lines of code, 67 lines of comments
Javascript: 361 lines of code, 32 lines of comments
XML: 1,073 lines of code, 47 lines of comments

Yup since the last update, 2k lines of code have been added, which includes the following changes:
* permissions revamp
* some of the user profile area (personal action log, showing the user's permissions)
* added [[ticket:xxx]] wikilink (two different implementations)
* a maintenance mode option to lock out users from the helpdesk but not admins
* improved UI in the admin area such that items can be toggled in Javascript if they're not applicable/available