Solution to Linux NTFS performance woes :- Bad performance / 100% CPU usage when using VirtualBox / VMWare and In general

Hey everybody!

On my dual boot system, apart from the system partition for Windows and Root+Swap partitions for Linux, I have one large partition of about 400 GB dedicated for storing my data which includes my shared Thunderbird email data folders, my documents, my videos, my music and what not. For obvious reasons, this partition is formatted with NTFS file system, for easy data sharing between Windows and Linux (BTW, If you really wanna know, I use Windows 8.1 Pro and Kubuntu 13.10, Kubuntu being my primary OS as of now :-) ). I was quite happy with my setup except for two very annoying problems (which use to happen in Linux Mint too that I was using previously) :

1. The Thunderbird installation use to lag too much on Linux, and sometimes on Windows too while starting up and reading multiple mails or accessing folders.

2. While using any Virtual Box or VMWare VM under Linux, I was getting pathetic performance and my host system use to hang a lot while reading or writing files inside the VMs. In system monitor i could see a lot of processor time going to the process : mount.ntfs. Also, in VirtualBox, whenever I tried merging a snapshot with the base image, the process use to hang and never complete.

High CPU usage by mount.ntfs

The Thunderbird problem was not that severe so my attention was solely on the VM problem and for some time I thought that this may be an issue with VMWare and Virtual Box. But even after upgrading to newest versions of both the software multiple times, this problem was never gone. And besides, on giving it a well deserved thought, I realized that the process mount.ntfs is not specific to Virtual Box or VMWare.

So basically, this seemed to me as an issue with the file system driver itself, namely NTFS-3G. I searched a lot on the net for a solution but didn't find any, there were only the same questions, that I was also asking. Frustrated, I decided to look into the official specifications and FAQ section of the driver developer's website (www.tuxera.com) and viola! I found the answers to all my issues with NTFS file system in Linux. Below are the exact steps using which you can also get great performance on NTFS drives under Linux:

1. Keep it Uncompressed! Period.

NTFS is a closed source file system and the NTFS-3G driver was created using some very sophisticated reverse-engineering techniques. Now, all the code revisions over the past few years have made it very speedy and bug free but still there are some grey areas where it cannot compete with the native driver as far as performance is concerned (and it is also not expected to. Remember, NTFS is not a preferred FS under Linux, it's there for compatibility with the Windows world).

Transparent Compression, is one such feature. Under Windows, you can compress a particular folder or even a whole drive using the native file system compression feature and it works great. The files are compressed and decompressed on the fly when you use them and you don't notice a thing. Performance optimizations have also been done by Microsoft to make it work seamlessly. But when working in Linux, all is not so hunky dory. While decompressing and compressing files, the NTFS-3G driver takes way too much CPU power and being a file system driver with more privileges, it hogs the system resources like a monster, uninterrupted for the most part. So the basic thing you can do to get about 10 times more performance is to decompress your drives that you share under both Windows and Linux. To do that, just right click the drive in Windows Explorer, select properties and uncheck the option  “Compress drive to save disk space.” and click Apply. In the next dialog that appears, choose “Apply changes to :\, subfolders and files” and click OK.

Remove drive NTFS compression under Windows

 If you have lots of files on the drive, this process can take some time so have some tea and snacks. This procedure will decompress the whole drive. If you don't want that, then at least decompress the performance critical folders on your drive like the ones where you have kept your VM virtual hdds (VDI, VMDK or VHD files). You can do that by right clocking on the folder, clicking Advance... button and unchecking the checkbox that says “Compress contents to save disk space”. This will improve the performance a lot and you will immediately notice the difference as soon as you will boot into Linux.

Remove NTFS folder compression under Windows

2. Enable Big Write Mode and Disable Last Access Timestamp Updation:

The NTFS-3G driver supports a flag called big_writes while configuring your file system in /etc/fstab or while mounting using mount command. What this essentially does is that it instructs the driver to write data to disk in larger chunks instead of on every single write instruction received by it. This helps a lot with throughput while writing/copying/moving large files and is in general good for small files too.

Similarly, NTFS has a feature of recording last access time of a file and this this is done every time the file is accessed, which adds up to the total time it takes to read from or write to the file. This can be safely turned off without causing any harm to the data.

To configure these options, below are the settings I use in my /etc/fstab file. You can use the same flags as in screenshot, other details will vary from system to system depending on how many drives you have and how you configure them.Basically, the highlighted items are the ones you wanna change in your config.

Configure big_writes and noatime mode in /etc/fstab

3. Disable mlocate/locate indexing of NTFS drives

mlocate or locate is a standard program under linux which can be used to search the file system quickly for a file or directory. It uses a high performance index of the file system,  generated and updated every day by the updatedb command. Usually, this is a scheduled activity by default on most systems.

The updatedb utility has some issues with NTFS file system, where even if a single file or folder is changed on the file system, it considers all the files and folders as changed and re-indexes everything on the drive. This obviously takes CPU resources and if the drive is compressed, the situation becomes more problematic because of the high CPU utilization by compression/decompression routines. This doesn't happen too much now-a-days it seems, probably due to updated versions of these two commands but still, changing a little configuration option for these commands can give you much better results.

The trick is to disable the index generation on NTFS file systems altogether. Usually, indexing is not required on NTFS and you can always go and search items using your GUI file managers if you need to. To disable it, edit the file /etc/updatedb.conf and add the entries ntfs and ntfs-3g to the "PRUNEFS=" line, like in the screenshot below. I am not sure whether ntfs-3g is needed or not but there is no harm in adding it so I add it nevertheless.

After applying all these tricks, my system has become so fast and responsive that I can finally use it without a hitch as my production machine for all purposes. Try these and let me know in the comments how it worked out for you.

 Cheers!

Truth About Internal Memory in Samsung Android Devices

Hey Everybody!

On 15th this month, I got my shiny new Samsung Galaxy Note II, GT-N7100, the international version or to be precise, the Indian version, in the color of my choice, Titanium Grey. I moved on to it from a 32 GB Galaxy S3,  basically for a bigger screen, much better battery, and the mighty S-Pen :) Needless to say, I am on cloud nine since then :D

But as always, you don't get everything of everything and this phone is no exception. My biggest gripe is the 16 GB memory touted by Samsung. It is not 16 GB actually. You only get 10.45 GB out of the box, and the rest is taken by : 

1. The calculation fiasco that virtually every company making storage devices on earth do, where they calculate 1 GB = 1 Billion Bytes instead of 1073741824 bytes, and 
2. The memory eaten away by Android OS and pre-installed software, basically your phone ROM. 

In addition, when you first boot your device, the initialization process also creates a few sqlite databases and other files on this memory reducing it even further.

In newer Android devices, I think the Samsung has stopped making different physical partitions (or memory chips) for ROM and internal memory. Instead, what they do is they divide the same physical memory chip into 2 or more logical partitions and then mount the partition with "ROM" contents as read only. If you are aware of the process of flashing a Samsung phone with ODIN, you might have found the word PIT file used often. PIT file actually defines the partition layout. That's why it is said that while doing normal ROM flashing, the re-partition checkbox should not be checked, it can wreak havoc if not done properly.

This layout provides them the flexibility where if the size of ROM contents decreases or increases, they can just re-partition the memory to adjust for it. This was a problem in previous devices, like the Galaxy S, which only got the Value Pack instead of an upgrade to Android 4.0 ICS, as their isolated ROM memory chip was not having enough space. Last, i checked around 60-70 MB was free in /system on my Galaxy S i9003.

Samsung doesn't release 32/64 GB versions of their phones in many countries like India many times. For example, in Philippines, 64 GB version of Note 2 is readily available but In India, only 16 GB is there. That's one of the reason why people resort to Rooting. Rooting actually provides a way to go over this limitation by using a technique called Directory Binding, which I will talk about in my upcoming posts. But this is one thing that I don't like about you, Samsung! I hope you are listening!

Solution: Bluetooth not working after upgrading to Ubuntu 13.04 (Raring Ringtail)

If you are like me, there are good chances that you love Ubuntu as an OS and there are even better chances that you have already upgraded to the latest and the greatest flavor of it, version 13.04 code named Raring Ringtail. Upgrading an OS is not similar to Installing a fresh copy, well, we all computer veterans know that. 

Upgrading almost always brings with it its fair share of problem.One such problem i recently had was a non working Bluetooth. In Ubuntu 13.04, the Bluetooth applet has been changed a little and its more sophisticated looking now. But somehow, on some computers, when you are upgrading from 12.04 or 12.10, the Bluetooth stops working. Whenever I was trying to send any file to my Samsung Galaxy S3 smartphone, i was getting following error :

Error : GDBus.Error.openobex.Error.Failed: Unable to request session

I researched a lot and couldn't find any proper solution for it. In the process, I came across a Bugzilla entry : https://bugs.launchpad.net/ubuntu/+source/gnome-bluetooth/+bug/1148033 .But here also, the bug is only listed, no official solution has been posted as of now. But going through the user comments I found that few people have commented a solution, to use a software called BlueMan a.k.a Bluetooth Manager.It is available in official repositories so you can install it by simply issuing the following command:

sudo apt-get install blueman

I installed it and started it. Immediately, I saw another Bluetooth icon in my notification area which had somewhat similar options to the "official" icon. So I thought that was it and tried to send a file from this new icon's Send Files option. Turns out, it was able to pair with the phone but not able to send anything. It was repeatedly getting stuck and then reporting that there was an error sending the file without any much helpful message. BAM!

This was testing my patience now and I cursed the Ubuntu team a little :) Then out of the clear, blue sky, I got an idea and I tried the Send File option of original Bluetooth icon. It worked!! But when I exited Blueman, the situation reverted to previous state.

So basically the solution was to keep Blueman running and using the original Bluetooth icon. It was less than perfect as I now had two icons in my notification area, but it worked.

But we tech-guys do not stop until we get what we want, at least as far as computers are concerned. So I dig further and finally found an option to turn off the blueman icon while still keeping it running. In the Blueman icon's menu, there is an option to access its Plugins. Use that, and disable the tray icon plugin. That will get rid of the extra icon. However, if anytime you need to get the icon back, you will have to re-install the utility. I personally don't know how to re-enable it without that. If anybody has some idea, let me know in comments.

Hope this will help some poor soul. :)

Happy Computing!

Changing Office 2007 Product Key without Reinstalling

Hello Everyone,
Once in a while, we all encounter sme strange, unknown problem. Think of it, you accidently got a counterfeit version of the Shiny new Microsoft Office 2007 and suddenly you got the "news" about it while downloading a template from Office Online, by the OGA system. So your conscience tells you :-D that it is wrong and you go an purchase a legal product key for yourself. Now comes the hard part! Where to insert this key? There is no visible place on UI from where you can change your product key. So now all you can think of is just uninstalling and reinstalling Office and all the other Service Packs etc. That is indeed a time consuming process and is cumbersome.
So here is the way to do it without reinstalling :
1. You have to be login as an administrator for this, as standard users cannot edit the registry directly. Apart from that, please back up your registry or create a manual restore point first as messing with registry can be quiet dangerous. :-)
2. Go to Run, type regedit and press enter. In Vista, press OK when the UAC prompt comes, if it is enabled.
3. Navigate to HKEY_LOCAL_MACHINE\Software\Microsoft\Office\12.0\Registeration.
4. Under this key, u'll see one or more subkeys with GUID type names (names made of random nos and words enclosed within { } )
5. In all of these subkeys, one by one click on them and check for the presense of these two values
(i) DigitalProductID - Type : Binary - Data: some long binary string written in multiple pairs of two characters .
(ii) ProductID - Type: String - Data: A product id made of numbers only in the form xxxxx-xxx-xxxxxxx-xxxxx.
6. Delete these values from each of these sub keys. Please be sure what you are deleting. If you delete something wrong, you might end up ruining your system.
7. Exit the Registry Editor and start any of the Office Applications. On starting, it will ask you automatically for the new Product key.
And thats that! Your office will turn to legitimate without reinstalling.
Enjoy! :-)

Introduction to Silverlight Development Environment

Hi All,
Its been a long time since i last posted, i apologise for that, if you are a regular visitor that is. So here is the first one for all you silverlight developers. I have seen many of the new Visual Studio 08 developers wondering around the silverlight technology and all the tools associated with it and how they are important and useful for a Visual Studio 08 developer. So here it is...

In simple words, Silverlight is a plug-in. Many people who have not begun development on Silverlight, confuse it to be a tool of some sort with a UI. I have observed that this notion usually comes from Flash developers/designers. In the case of Flash, the tool used to build applications is called Adobe Flash and the plug-in required to run Flash applications is called the Flash Player.

On the Microsoft end, Silverlight is the name of the plug-in - which contains all the runtime elements required to render the UI of your applications and execute the code of your applications. The Silverlight runtime is installed one-time on your computer. You could install this be going to www.microsoft.com/silverlight or by visiting any page that uses Silverlight and clicking on the "Get Microsoft Silverlight" icon when prompted.

Installing Silverlight has enabled your browsers to view Silverlight content. For development of Silverlight content, you require a set of tools.

These can be categorized as designer and developer tools.The designer tools you'll require is primarily Microsoft Expression Blend. Expression Blend helps you create Silverlight projects and WPF projects. WPF stands for Windows Presentation Foundation, which is part of .NET Framework and is a superset that is used to create rich desktop applications. Silverlight projects created in Expression Blend follow the same project structure that developers working with Microsoft Visual Studio are familiar with.

Expression Blend is primarily used by designers to:

Create the front-end design for your application: This is created using vector elements from the toolbox such as Rectangle, Ellipse, Line, Path, MediaElement, Canvas, etc. and using the properties pane to set various properties such as Fill color, Fill type (Solid, Linear Gradient, Radial Gradient, etc.), Stroke color, Width, Height, etc. Each element has several properties that can be applied to it and most of these can be set from the property pane directly.
Organize elements: Elements added to the application can be named to help in identifying them later on. Multiple elements can also be grouped. Grouping automatically creates a Canvas element (with the width and height of the boundary of elements) and groups the elements as child elements of the newly created Canvas element. (Remember: Canvas is the only container element in Silverlight 1.0). You can also use Expression Blend to change the z-order of the elements. This basically means which element resides on top of which other element - something that is done automatically based on the order of the elements - and can be changed by either changing the order itself or by changing the Canvas.ZIndex property. Animation: Animations are created using Storyboards. It is helpful to switch to the Animation workspace (from the Window menu) to help in creating Storyboards. Once a Storyboard is created, timelines are recorded using the Timeline Recording feature. Microsoft Expression Blend is currently available in two versions - Expression Blend v1 was released in May 2007 and only supports WPF development. For Silverlight development, you will need Expression Blend v2 which is currently under development and preview releases are available from www.microsoft.com/expression. Look for the latest preview release and be sure to download Blend 2!

All design created in Expression Blend is automatically translated (as you add elements) to XAML. XAML is an XML-markup to describe the UI. The Silverlight plug-in parses the XAML when a Silverlight project to render the UI in the browser. Anything you do in Expression Blend creates the XAML in the background. You can also switch to the XAML view in Expression Blend to view the markup. This is akin to creating HTML pages using a tool when the tool is generating HTML markup in the background. XAML is a common UI description language across WPF and Silverlight, though the number of XAML elements you can use in WPF is much larger than Silverlight 1.0. Expression Blend automatically changes the available toolbox elements based on whether you are creating a WPF project or a Silverlight project.

Microsoft Expression Design is the other tool you can use in the Silverlight development environment. Expression Design is used for much finer elements of design, with support for layers, various brushes, effects, path transformation tools, etc. It is a vector design tool that can help you create design assets for use in Silverlight. Say that really cool looking button with extra-fine detail. Since many of the tools and shapes you use in Expression Design do not have a straight-forward mapping to XAML (which is what Silverlight can understand), you export your design to XAML once it is ready. The export process looks at the design, rasterizes (converts to images) stuff that is incompatible with XAML, converts vector shapes to paths or other supported elements and provides XAML that is compatible. Such XAML can then be included into Expression Blend to add further animation or interactivity.

Microsoft Visual Studio is what developers use to add event-handling code that can then call back web services or can trigger other actions based on user-input. Every element has event handlers for mouse, keyboard or ink input and these are handled in code - JavaScript is used as the programming language for Silverlight 1.0.

As a developer, you can use notepad for authoring both the XAML and the JavaScript and for embedding the Silverlight object on to the HTML page. However, Blend and Visual Studio provide a much richer development experience and most of the code for embedding the object is automatically generated.

To get Visual Studio to understand the XAML semantics, it is worthwhile to install the Silverlight SDK available on www.silverlight.net. The Silverlight SDK also comes with some sample code and quickstart examples, as well as the all-important documentation. The documentation contains a full reference for all objects, methods, properties and events and also several basic articles to get you started with the concepts.

So in summary, Silverlight is the plug-in that you or your end-users require to install to enable your browser to render Silverlight content. Expression Blend and Expression Design are part of the Microsoft Expression Studio which enable designers to design Silverlight projects. Visual Studio is the developer tool for developers to write the required code for the applications. Silverlight SDK gets you started with required files, project templates, documentation and samples.

Really simple, right? Happy coding!

Source : Pandurang Nayak's Blog

Microsoft .NET - 10 Great Resources for One Great Technology

.NET is the hottest thing these days in developer circles specially now with the release of Visual Studio 2008 (Orcas) from Microsoft. It can take advantage of all the new features introduced in Windows Vista and the 2007 Office System. The .NET technology specially became very popular when the 2.0 version of it was released. The response was huge and lakhs of sites are using this technology to deliver rich content. Silverlight technology and the Microsoft Expression series of products added to it and briged the gap between the designer and developers. Silverlight specially makes it very much easy to created RIAs(Rich Internet Applications). So .NET is not a small thing, its includes a truckload of components to learn and implement be it the disconnected data model of ADO.NET or the highly scaleble server side programming model of ASP.NET. Almost any type of application can be developed in the .NET Framework including Desktop Application, Class Libraries, Windows Services, Web Applications, AJAX enabled Web Applications, Web Services, Data enabled Web and Windows Applications etc. The only thing that the .NET framework is not suitable for are low level device drivers and stuff like that.
.NET framework is platform independent, provides language interoperability, cross language integration, integration with older technologies like COM and Windows DNA, a large number of languages to program in, and lots more. Now to leverage all of this stuff, you need to have proper resources with you. Books alone are not good enough.
Fortunately there are a lot of websites that have good information and training material on .NET that one can use. I am posting here the ones that i use the most:-

1. http://msdn.microsoft.com/
The very first place you should look at for anything related to .NET
2. http://www.asp.net/
A good place for learning material related to ASP.NET and related technologies like AJAX.
3. http://www.codeplex.com/
A community driven forum for open source projects on .NET
4. http://www.neowin.com/forums
A good place for any query regarding computing. The members are very helping and you'll get response almost always.
5. http://www.codezone.com/
Codezone connects you with top-rated user groups, meetings, community sites, blogs, forums, events and breaking news. It’s a place for developers to share and promote top-notch content and community resources.
6. http://forums.msdn.com/MSDN
The MSDN Forums allow you to search a growing archive of technical questions and answers.
7. http://msdn2.microsoft.com/en-us/events/default.aspx
The MSDN events and Webcasts homepage is a nice place to get live webcasts and videos from Microsoft employees themselves.
8. http://channel9.msdn.com/
Channel 9 provides direct, unfiltered, and open access to Microsoft’s cockpit. Meet the people who think up and build Microsoft’s technologies, interact with your peers, share your ideas and applications.
9.http://msdn.microsoft.com/newsgroups/topic.aspx?url=/MSDN-FILES/028/201/317/topic.xml
Microsoft .NET Development Newsgroup. Get involved in discussions on various aspects related to .NET Development here with MVPs and Microsoft Employees, as well as other people like you.
10. http://www.learnvisualstudio.net/
A great place to find video tutorials related to .NET technologies. Although the site is paid, you can get many free videos there as well as signup for the monthly newsletter for getting free videos every month.

So thats all for now. Whenever i get more resources, i'll update this list (as well as its title :-) ). Hope these resources help finding your way in world of Microsoft .NET. :-)

{Heros Happen Here} The Mega Launch Event!

Good news for developers and IT pros working on Microsoft Technologies, Microsoft is organizing a mega launch event for 3 of its greatest developer and IT pro tools, namely Microsoft Server 2008, Microsoft Visual Studio 2008 and Microsoft SQL Server 2008. The event is taking place at 3 places in India. The schedule is :

Bangalore: March 4, Rex Theatre.

Mumbai: March 5, Regal Theatre.

Delhi: March 7, PVR Priya Theatre.

The launch will be featured on the 75 mm screen of cinema and will include a free showing of the movie Gladiator. Whats more, every participant of the even will be given free copies of all these software as a promotion.

Everyone who is intrested in attending the event can register on http://msevents.microsoft.com/?Culture=en-in or from http://www.heroeshappenhere.co.in/. You need to have a hotmail or windows live id for that.

Also note that laptops are not allowed in these events and parking is very limited, specially for two wheelers. So plz be aware. See you there! :-)