Out of memory exception using Newtonsoft.Json package

Newtonsoft.Json package is probably one of the most essential packages in .NET software development. For those of you, not knowing what it does. It takes care of object serialization to JSON notation and deserilaization from JSON notation. I have used this package in numerous projects since its inception and I can only say great things about it.

However, as a part of one of our product’s GDPR compliance upgrade, I encountered an interesting undocumented feature. After object serialization a w3wp.exe process running the application pool for our product, started consuming 100% CPU capacity and hogged so much memory, we experienced “Out of memory exception” in a matter of minutes.

Since our product still uses .NET 3.5 (we are planning an upgrade to 4.7.2 shortly), tasks and parallel library are not native. We are using Microsoft’s TaskParallelLibrary package to circumvent framework deficiency. Hence, at first, I was dead sure that the library was a source of this issue. Specially, as we were doing serialization in an asynchronous method. After removing creation of new task, I was surprised it was not the case.

The object we wanted to serialize was a more complex derivative of this:

public class OurObject {

    #region Fields
    
    private int? _id;

    #endregion

    #region Properties

    public int Id {
        get {
            if (!_id.HasValue) throw new CustomException("some message");
            return _id;
        }
        set { _id = value; }
    }

    #endregion

}

The easiest way to serialize an instance of this object would be to do something like:

var instance = new OurObject();
var json = JsonConvert.Serialize(instance);

Except, this throws a CustomException, as Id property is not set. Newtonsoft.Json package documentation and StackOverflow answers offer a solution using serialization settings:

var instance = new OurObject(); 
var json = JsonConvert.Serialize(
    instance, 
    new JsonSerializerSettings { Error = (se, ev) => { ev.ErrorContext.Handled = true; } });

This works as expected. It ignores exceptions thrown by a serialized object instance. Yeay!

Not so fast. When using above code as part of web application, it will cause your application to hog all available CPU power and consume as much memory as possible. Promptly. Yikes! Surley, not something you would want in a production ready environment. Running a debugger revealed issues with this solution. Whenever a serialized object raised an exception, because of our serialization setting, exception went by unhandled. This in turn put stress on servers CPU and caused a memory leak of Mt. Everest.

The bad thing is, that at the point of my writing, there is no option, to tell JSON serialization engine to handle all exception raised and not just mark them as handled. I guess what you could do is create new properties for each property causing you a headache and then decorate it with JsonPropertyAttribute accordingly, but in our case, that would mean changing every property in an object (and there were plenty). What I ended up doing was that I converted an object to DataTable (we use it for ADO anyway) and serialized that. Worked like a charm.

Web developer: Why 2017 feels exactly like 1997?

I don’t know how many of you, dear readers, remember still how it was like being a web developer in late 90s? You know, the time when not every kid knew how to do web-sites. The time of Geocities, Angelfire and Lycos. The time without Google (well, nearly). The time before cross-browser javascript frameworks and no real support for CSS2. These were the times when 5% of your work presented doing the actual web site and 95% of time was spent on tweaking HTML, CSS and JavaScript to actually work in Netscape 4 and IE 5. And in the end, you somehow always ended doing layout with tables in tables in tables… Yeah. You didn’t want to be THAT guy.

But, with web becoming “the thing” and web sites started to blossom, we got Netscape …uhm… 4 and IE6 and CSS2 support improved (yeah, right) and first semi-useful cross-browser library came to life. It was called cross-browser and surprisingly enough it still online. As funny as it looks today, this was the first library where you didn’t have to pay attention to browser specifics. It gave us at least a glimmer of hope that future is going to be better and bright…

Fast-forward 20 years. Internet Explorer and Netscape are a thing of the past. Chrome, Firefox, Edge and Safari browsers are now in. We have full CSS2 support (well, very nearly) and so many cross-browser javascript libraries that we can’t even name them. Yet, working on my side project TimeLoggerLive, I started to wonder. Is it really that different? I mean, sure, new technologies are out (HTML5, CSS3, Angluar7000 etc.), but has things actually changed for web developers?

CSS3 initial release was in 1996 and HTML5 standard was in preparation since late 2000s. Yet, all these new browsers that we switched to because of promised support for “the latest and greatest” standards still don’t support either in full. Worse even. Like back in 1990s, each and every browser implements the standard differently. You can imagine the confusion.

In case of TimeLoggerLive, my kryptonite is a property contenteditable. The property’s functionality is awesome. By setting it’s value to true, you are supposed to be able to edit content of any HTML tag, provided the property is set on that tag. Handy. Except, it does not work on all tags in IE and Edge browsers, Firefox has some strange behavior, if you use it on empty cell and Chrome, which offers the best implementation of it, for some odd reason, distorts column width.

I checked one of my favourite pages CanIUse.com and it is marked as full support across all browsers, but Opera mini. However, there is a “known issues” section, where it is explained that IE browser does not support contenteditable property in following tags: TABLE, COL, COLGROUP, TBODY, TD, TFOOT, TH, THEAD, and TR. To avoid this, one needs to implement a DIV tag into each table cell. Groovy. Except, when you implement DIV tag, suddenly all browsers start showing border around editable content, which leads to more nasty CSS hacks.

Yes. It feels exactly as 1997.

Quick tip: Setting Oracle client collation

This week one of our clients experienced an interesting problem. Data obtained from ORACLE database did not display unicode characters. They were either replaced by ‘?’ or some other character.

This happens for one of two reasons (or in worst case scneario both). Either your database has wrong collation or your ORACLE client does. The former is a bit difficult to fix, as you will need to change database collation and existing data. The later is a bit easier. Here is how you do it:

ORACLE client 8.x

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\ORACLE]
"NLS_LANG"="SLOVENIAN_SLOVENIA.EE8MSWIN1250"

ORACLE client 11.x

[HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\KEY_OraClient11g_home1]
"NLS_LANG"="SLOVENIAN_SLOVENIA.EE8MSWIN1250"

I had to set Slovenian WIN1250 ecnoding and this is what sample does. More languages and options can be found in ORACLE documentation here and here.

Make it pink

There is always interesting time, when new project is on the board. This time it was a mobile application and me and my coworkers were tossing ideas left and right. So a discussion got a turn to what color scheme should user interface use. Mostly out of fun (and some out of envy), I claimed: “Pink. It should be pink.”

Back then, I thought nothing of it. The statement was meant as a joke and the color was picked based on what would be most inappropriate for this project. Coworkers took it as a joke as well. As a (more and more annoying) joke I threw at them every time, the discussion turned to UI for some reason. But, as things go, the project was ready to be presented to prospective customers. Naturally and understandably, sales pushed for customer company brand color scheme for each demo. Although coworkers did a wonderful job on the project, it still took some fiddly last-minute work to sort the new color scheme and base it on a theme.

At that point, I started figuring out, that as dumb, as my “idea” was, it would be beneficial to the project, if the UI really did use pink color. Reasoning is pretty trivial. When creating a project from scratch, there is usually so much work with basic mechanics, feature implementation and testing, that UI implementation is almost an afterthought. Hence, to solve glitches as quickly as possible, we, developers, tend to use line of least possible resistance and do something stupid like specify color or font inline. I admit, I did it numerous times on other projects. I know it is wrong. Heck, I knew it back then. But I just justified it with the fact that there just wasn’t enough time.

Now, if I made one of those project pink, I am pretty sure, I would do everything in my power to not use cheap corner cutting techniques and to make darn sure that I could swap color scheme faster than you can say cookie with mouth full of, well, cookies. This applies to all other areas as well. Want to make sure that you do not have static text in your project? Use upper case text for everything. Want to write data somewhere? Store it to file on floppy instead.

Thus, the next time you are about to do something, make it pink!

 

 

API pr0n

I doubt that anyone serious about tech could avoid hearing at least a little bit about NPM “disaster” in recent weeks. Some say, it is developers ego, that caused entire ecosystem to crash down. Others blame NPM for caving in to capital instead of open source. But the problem lies elsewhere.

In recent years, a phenomena started to occur in software development cycles, that I like to call API porn. It is caused by a fact that services, such as npm, bower, nuget and such, make it as easy to add APIs to projects as it is snapping your fingers. Don’t get me wrong, I love code reuse, but this has crossed every possible edge of reason.

Let me explain why. One of most challenging and difficult things for developer to do is an API. Why? It needs to be open just enough that another developer can use it, but not in irregular way. It needs to capture every single edge case, just for the fact that a memory leak does not make you reboot your app every 15 minutes. Or, that your application does not crash, when your users are entering unicode characters. Talking from experience, this takes a lot of thinking, tweaking, debugging and testing.

In that retrospect, there are more than 500.000 APIs on NuGet alone and I am guessing NPM can beat that without a sweat. Half a million APIs.  Have developers became so much better at writing APIs or is it that we publish just about everything nowadays? As much as I would like to believe it is former, quantity wins.

Back to NPMs little problem. The package that did most “damage” was left-pad package. Let that sink in for a moment or two. Left-pad package is nothing fancy (no offense to the author). It is a simple JavaScript function that adds custom padding in front of a given string.  Yes, that is correct. We have come up so far, that people actually think, it is better to include 3rd party left padding extension than it is to write your own. Specially, when it will take about 35 minutes, inlcuding a lunch break. And this same people are now blaming developer and NPM when their projects don’t compile/deploy? Somehow reminds me of people that copy and paste first StackOverflow solution (without reading) and then complain it does not work.

This must stop. Now. I know, that package managers open entire world of possibilities, but “with great power comes great responsibility” and as it was obvious from NPM crash, most developers just cannot be trusted with it.

Quick tip: Cache busting in ASP.NET revisited

Anyone that tried to cache static files eventually got to a point where cache caused more problems than it solved. After all, telling all your users to press Ctrl + Refresh in their browsers is not exactly how one should do things on the web. Two years ago Mads Kristensen presented us with a solution in his article Cache busting in ASP.NET. The solution uses Fingerprint class, that basically updates cache object every time a static file is changed.

All fine and well. So why do I jab about it now? The solution, in my opinion has two glitches:

1. It is tightly bound with URL Rewrite IIS module.

2. It always links static content to root URL.

So, without further ado, I present you with “upgraded” solution that avoids both issues and works with relative URLs (relative to application URL anyway). In-page usage remains the same.

using System.IO;
using System.Web;
using System.Web.Caching;
using System.Web.Hosting;

public static class Fingerprint
{

	public static string Tag(string rootRelativePath)
	{
		if (null == HttpRuntime.Cache[rootRelativePath])
		{
			var absolute = HostingEnvironment.MapPath("~" + rootRelativePath);
			var dateLastWrite = File.GetLastWriteTime(absolute);
			var result = VirtualPathUtility.ToAbsolute(
				string.Format("~{0}?{1}", 
					rootRelativePath, 
					dateLastWrite.Ticks));
			HttpRuntime.Cache.Insert(
				rootRelativePath, 
				result, 
				new CacheDependency(absolute));
		}
		return HttpRuntime.Cache[rootRelativePath] as string;
	}

}

This code is also available as gist.

Memory leaks in .NET?

In olden times, when I did C and C++ development, memory leaks were a common thing. From time to time I still wake up from nightmares in which I debug, trace and try to locate and plug memory leaks (of course, as this is nightmare, I fail to do either, or it wouldn’t be a nightmare). Thankfully, in unmanaged code, memory leaks can be spotted in a debugger (or a plain old segmentation fault) and if you are paying attention, you can plug the leak right after you made it.

Then came Java and .NET, JIT compilers and a great thing called garbage collector. And we were told, we can forget about memory allocation, deallocation and pointers as a whole. We now have a garbage collector that will do this for us. And it does, if you know how to use it. But now, we became so reliant on garbage collector, that we just assume it will do everything for us. These days, it seems developers think that just because application is written in .NET it will somehow obtain immunity to memory leaks.

Boy do I have news for you! It won’t. Just last week, I spent most of my time debugging, tracing and removing a memory leak that caused otherwise stable windows service to throw OutOfMemory exception in 90 minutes. As .NET code is managed, tracing the leak gets tricky and the fact that code was not mine only made it worse.

If you write a one-off trivial application, memory leaks are nothing to worry about, as garbage collector will release the memory, when application is closed. And since application runs only short period of time, unless you do something completely moronic, you are fine and users might not even notice.

But, as soon as you start writing serious applications and applications that have to run 24/7 (e.g. Windows Service), things change. Any serious application has to assume, that, if object is not disposed correctly, it will accumulate in memory until the application is closed or it crashes. Preferably on Friday at 3pm when you are about to head for your well deserved vacation.

So what can you do? Personally, I stick to one simple rule that removes a risk of good portion of possible memory leaks. Any object that implements IDisposable interface has to be either encapsulated in using statement or disposed explicitly. Preferably in finally statement of try-catch block. It also helps to go the extra mile and remove reference to objects used as soon as possible and thus allow garbage collector to do its job properly.

And our memory leak? With great cooperation from our customer and some good fortune, we obtained enough information to locate the leak. It was caused by opening a Stream object about 1 million times without ever closing it.

Absolute minimum you need to know about storing passwords

In 2013 Forbes reported that approximately 30,000 sites across the globe are hacked each day. Yet, for some odd reason, in 2015, I still see web applications that employ questionable password storing mechanisms e.g. storing passwords in plain text or just hashing them with md5. With 30,000+ sites hacked a day, there is a strong chance that your site will be hacked next.

[Read the rest of this entry…]

Mobile app development: A pain in the b…

You kn0w, I am amazed by abundance of videos and tutorials out there that show you, how “easy” it is to create a mobile app. Sure, creating a “Hello world!” example with a few fields and a progress bar is not difficult, but that is hardly a representative app. In real world, you would want user to authenticate, you would want to store some user data (you know, in case he or she decides it is time to wipe the phone and then regret the decision), you would want to offer some payable services to those users and differentiate between free and pro users etc. etc. Yet, you don’t see tutorials for that. And you know why? Because those things are hard to do right.

[Read the rest of this entry…]

Advice to young developers

In last few months I have been assigned as a mentor to young developer who wanted to work as a student for our company. This is my first foray in being a mentor to someone, let alone a prosperous young talent. The first thing when I was told about this, panic kicked in. But then I started thinking about the time when I was a young developer starting out. What would I want to know? What would I want to learn?

Teaching someone a specific technology is pointless. So what could I teach my apprentice that he can, hopefully, take with him for entire lifetime?

I narrowed the list down to four points. So, to you young-lings, I’ve got this advice to offer:

  1. Think first, code later.
  2. Learn how to use version control and bug tracking software.
  3. Listen to your elders.
  4. Question everything.
  5. Get out of your comfort zone.

[Read the rest of this entry…]