Tag Archives: OS

Top 5 New Features in SQL Server 2017 (that I care about)

New Year, New Database Engine

http://icons8.com/
This is either a database or a stack of licorice pancakes.

It’s finally here! A few days ago as of this writing, SQL Server 2017 was released for Windows, Ubuntu, RedHat and Docker. There are a lot of new things in SQL Server 2017 from Python support in Machine Learning* to better CLR security. But I thought I’d narrow down the list to changes that I’m most interested in.

1. sudo Your Way To A Better SQL Server

SQL SERVER IS ON LINUX NOW! No surprise this is the first thing on my list as I keep going on and on about it. But it’s really here!

Installation is a cinch, especially if you’re a Linux or Unix person. Curl the GPG keys, add the repository, apt-get (if you’re on a real Distro) the installer and run the setup. It’s really that easy.

If you want it even easier, then check out Docker. Slap it in and go!

All the main features are there, the engine (of course), agent, Full-text search, DB Mail, AD authentication, SQL command-line tools etc. Pretty much everything you need to get going on Linux even if you’re in a Windows-dominated environment.

2. It’s Like A Car That Repairs Itself

So this is a big one. With automatic tuning on, SQL Server can detect performance problems, recommend solutions and automatically fix (some) problems. There are two flavors of this, one in SQL Server 2017 and one in Azure; I’ll be talking about the one in SQL 2017 here.

Automatic plan choice correction is the main feature and it checks whether a plan has regressed in performance. If the feature is so enabled, it reverts to the old plan. As you can see below, a new plan was chosen (4) but it didn’t do so well. SQL 2017 reverted to the old plan (3) automatically and got most of the performance back.

Image Source: Microsoft

I’m sure that Microsoft will be expanding this feature in the future and we can expect to see more from this. Azure already has automatic Index tuning in place, so we’ll probably see that in the On-Prem version eventually.

3. Indexes That Start, Stop And Then Start Again

This is a feature I didn’t know I needed. Basically, it allows an online index rebuild that has stopped or failed (say, it ran out of disk space) to be resumed. The index build fails, you fix whatever made it fail, and then resume the rebuild, picking up from where it left off. The rebuild will be in the ‘PAUSED’ state until you’re ready to RESUME or ABORT it.

Code:
-- Start a Resumable index rebuild
ALTER INDEX [NCIX_SomeTable_SomeColumn] on [dbo].[SomeTable]
REBUILD WITH (ONLINE=ON, RESUMABLE=ON, MAX_DURATION=60)

-- PAUSE the rebuild:
ALTER INDEX [NCIX_SomeTable_SomeColumn] on [dbo].[SomeTable] PAUSE

/* If you'd like to resume either after a failure or because we paused it. This syntax will also cause the resume to wait 5 minutes and then kill all blockers if there are any. */
ALTER INDEX [NCIX_SomeTable_SomeColumn] on [dbo].[SomeTable]
RESUME WITH (MAX_DURATION= 60 MINUTES,
WAIT_AT_LOW_PRIORITY (MAX_DURATION=5, ABORT_AFTER_WAIT=BLOCKERS))

/*Or if you just want to stop the whole thing because you hate unfragmented indexes */
ALTER INDEX [NCIX_SomeTable_SomeColumn] on [dbo].[SomeTable] ABORT

This is also great if you want to pause a rebuild because it’s interfering with some process. You can PAUSE the rebuild, wait for the transaction(s) to be done and RESUME. Pretty neat.

4. Gettin’ TRIM
A cute Sheepy.
There hasn’t been a picture in a while and I was afraid you might be getting bored.

This one is kind of minor, but it excited me a lot because it was one of the first things I noticed as a DBA that made me say ‘Why don’t they have a function that does that?’ (note the single quotes). TRIM will trim a string down based on the parameters you provide.

If you provide no parameters it will just cut off all the spaces from both sides. It’s equivalent to writing RTRIM(LTRIM(‘ SomeString ‘))

Code:
SELECT TRIM( '.! ' FROM ' SomeString !!!') AS TrimmedString;

Output:
SomeString

5. Selecting Into The Right Groups

Another small important change. In previous versions of SQL Server, you could not SELECT INTO a specific Filegroup when creating a new table. Now you can, and it uses the familiar ON syntax to do it.

Code:
SELECT [SomeColumn] INTO [dbo].[SomeNewTable] ON [FileGroup] from [dbo].[SomeOriginalTable];

Also, you can now SELECT INTO to import data from Polybase. You know, if you’re into that sort of thing.

Honorable Mentions!

Here are some things that caught my eye, but didn’t really need a whole section explaining them. Still good stuff, though.

Query Store Can Now Wait Like Everybody Else

Query store was a great feature introduced in SQL Server 2016. Now they’ve added the ability to capture wait stats as well. This is going to be useful when trying to correlate badly performing queries and plans with SQL Server’s various waits.

See the sys.query_store_wait_stats system table in your preferred database for the deets. Obviously, you’ll need to turn on Query Store first.

Con The Cat With Strings

Just as minor as TRIM in some people’s books, but this is a great function for CONCAT_WS’ing (it’s not Concatenating, right? That’s a different function) strings with a common separator, ignoring NULLs.

Code:
SELECT CONCAT_WS('-','2017', '09', NULL, '22') AS SomeDate;

Output:
2017-09-22

Get To Know Your Host

sys.dm_os_host_info – This system table returns data for both Windows and Linux. Nothing else, just thought that was neat.

Get Your Model Serviced

Not something I’m jumping for joy about, but it is a good change of pace (I think). No more Service Packs, only Zuul… er… just Cumulative Updates. Check out my article on it if you want to know more about all the Service Model changes.

And Lots Of Other Things

Of course, there’s hundreds other things that are in SQL Server 2017 and related features. I didn’t even touch on the SSIS, SSAS, SSRS, MDS or ML stuff. Check out the shortened list here, broken down by category. Exciting new toys!

-CJ Julius

* Seriously, how do you release a Data Science platform and not include Python? That’s like releasing a motorcycle with only the rear tire. Yes, you can technically use it, but you you’re limiting yourself to a customer base with a very selective skill-set.

Simple Database Inventory Manager 2.1

sdim_ver_2-1_trunc_275x400

A while back, I took on a fairly big project; build an database inventory manager that did the following:

  • Dynamically gathered information on SQL Instances/Databases
  • Dynamically gathered information on the OS underneath
  • Compiled and organized this data in a single repository
  • Provided a client GUI front-end for ease of use.
  • Was built on free and already available tools.

When I initially finished Inventory Manger 1.0 I wrote a 5-part series that took the user through the steps of how I built what I did and how everything worked. This was a good way for me to iron out details and also provide some documentation along the way. As the months have progressed I added updates to the code and altered the posts as necessary.

Then everything went silent as I moved on to other things, but I was constantly going back and adding new features and options. During this interim, I was not able to get the code updated on GitHub, and thus it fell behind. Also, it had morphed into its own beast, moving out of “project” and more into a standard “software” mode. To signify its new direction, its name is now Simple Database Inventory Manager™.

As such, I have added the new 2.1 version to its own GitHub project (previously it existed in the SQL code section there with other unrelated snippets) and will update it accordingly. You can just download the new files and overwrite the old ones to get the new version. Spiffy, right?

db_datapull_frontend_v2-1
Look! 200% more buttons!™

I will keep the old 1.x version in the original repository so it doesn’t break links for the walk-through articles.

All that said, what fabulous prizes are in the new version? Glad you asked!

CMS Integration

Now you can point to your Central Management Server and the Simple Database Inventory Manager™ will pull a list of servers and instances from there. It will then go through all of them recursively, and pull whatever data you want back.

In DB_DataPull.ps1, you can switch this on or off with the -UseCMS switch. If on, then you will need to specify the CMS Server with the -CMSServer ‘SOMESERVER’ parameter.

If you use the -UseCMS option, this will delete all data from the repository tables and repopulate them based on what you have in the CMS. This is backwardly compatible with the old system in that if you don’t use the option (off by default) then it will continue to use the Server and Instance List you provided manually.

Fresh, New Buttons

‘Services’ has been folded into the Inventory heading and Other Info is no more, replaced by the Reporting section (below).

Composite buttons have been created to give better side-by-side information from the GUI. Server\Instance and Instance\DB buttons have been added to do this. I think their names are pretty self-explanatory. Click on them. See what happens.

Reporting

Kind of. A bunch of stock reports were added to the DB_DataPull_FrontEnd.ps1 under the Reporting heading. These rely on Views in the Reporting Schema that build customizable information you want returned.

I’m going to leave this schema (Reporting) pretty much alone for now, so users can create their own views and then use those as datasets if you want for SSRS. You can use SDIM™ as the basis for reports that refresh as often as you want to run DB_DataPull.ps1

Here’s a quick list of the out-of-the-box reports given in the GUI:

Servers Grouped by OS and Service Pack – A count of all Server OS names and versions.

Instances Grouped By SQL Version – A count of all Instances at a specific server version (ignores Service Pack)

Instances Grouped by SQL Version, Edition – A count of all instances by SQL Version and Edition (ignores Service Pack)

Instances Grouped By SQL Version, Edition and SP – A count of Instances based on SQL Server Version, Edition and Service Pack

Bug Fixes/Minor Enhancements

  • Fixed an incorrectly spelled column.
  • Fixed incorrect calculation of Server number.
  • Added extended properties to all tables.
  • Added header at the top of all Views and Procedures

And All the Rest

This (Simple Database Inventory Manager™) is of course provided free of charge, use-at-your-own-risk. There is no warranty either expressed or implied. If SDIM™ burns down your data center, uninstalls all your favorite toolbars and ruins your best pair of dress socks, I’m not at fault. Remember to back up your databases!

And if you’ve skipped over everything just to get to the link, here it is: SDIMS v2.1

-CJ Julius

(Almost) Everything is Going Open Source Now… and I LOVE it.

Why can't we be friends?
Why can’t we be friends?

While I’m putting together my big update on Inventory Manager, I thought I’d take some time to throw confetti into the air. There may be some excited clapping as well. I warned you.

I largely see myself as platform-agnostic. While I think that certain companies do individual products well, I also believe it’s fair to say that none of them do everything well. I use Android phones and Apple tablets, Linux for home (mostly) and Windows at work. Heck, I’ve got a Roku and a Chromecast because they both do things that the other doesn’t.  I’m all over the map, but all over the map is a great place to be, especially in the tech industry now.

Despite all of this, I have to admit I am partial to Free Open-Source Software (FOSS). Give me a choice between Ubuntu and Windows, and all other things being equal, I’ll choose the Debian-based option. I’ll admit my biases.

So, when MS started moving in this direction I was happy. I wanted to see this trend continue, and boy has it. First of all…

1. .NET Core is now running on Redhat.

When Microsoft announced that .Net was going open-source, I was cautiously optimistic. I’m not a big .Net coder, but I could see the benefit and was hopeful that MS would continue down this path.  This lead to some cool things that I thought I’d never see in a million years, like .Net running on Redhat.

There’s understandably some cynicism about Microsoft’s true intentions, as well as their long term goals, but this is the cross-over that I’ve been wanting to happen for a while. Blending the strengths of RHEL with .NET on top is a great start. If the .NET development platform can be ported, why not parts of the Windows Management Framework? We could even one day see…

2. Powershell on OSX and Linux.

I didn’t always like Powershell, in fact prior to Powershell 3, I just referred to it as PowerHell. Since 4.0, however, it’s no secret that I’m a fan; one look at my github will tell you that. I like its logical approach to (most) things and that it works for simple scripts quite easily, while being a powerhouse (no pun intended) behind the scenes.

Sorry, THIS is the coolest thing ever.
This is the coolest thing ever.

This shell coming to OSX and Linux will be a boon for both systems. While I am, and will probably always be, a bash scripting guy, Powershell in Windows just makes everything so gosh-darn easy. If I could whip up a PS1 script with a few imported modules and attach it to a cron job with ease, then I think everybody wins,  mostly me. But, if I decide that I want to use bash instead, that’s okay because…

3. Bash is running on Windows.

This isn’t a one way transition. Microsoft is making a trade, bringing one of the most widely used shells to Windows. This not only makes scripts more portable, but also knowledge.

Have some ultra-fast Linux bash script that works wonders? Super, you now have it Windows, too. Wrote a script to do some directory work in Powershell? Great, you now know how to do it in Linux.

You can't tell me that isn't the coolest thing ever.
I’m sorry, THIS is the coolest thing ever.

There are very few downsides to this, other than the obvious security issues and that it isn’t truly a stand-alone shell (it’s part of Ubuntu on Windows). In any case, it allows interoperability  between software from different systems. This is great now that…

4. SQL Server is on Linux.

This isn’t technically going open source, as it will run inside a container, but the idea that this will now be possible and supported is like something out of my greatest dreams.

I have a maybe-controversial opinion that SQL Server is the best relational database system out there. For all its faults, I’d rather use SQL Server 2005 SP1 than Oracle 12c. Just the way I feel, and for reasons I won’t go into here. I hope the things I like about SQL Server translate to the Linux environment.

The fact that Ubuntu is supporting this with Microsoft is great. I can’t wait to use my favorite OS with my favorite database engine on the same system.

Last thoughts

There are other items I’ve glossed over, but these are the big ones to me. Soon, we will be able to run SQL Server on Ubuntu Linux with cron jobs executing Powershell for a .Net application that resides on an RHEL box. *excited clapping* (I warned you.)

It’s a great time to be in the tech industry.

-CJ Julius

 

Creating a Simple Database Inventory Manager with Powershell – Part IV: GUI Front-End

Last Time, on Inventory Manager…
Now we make things pretty.
Let’s make things pretty.

Our data pull script has run, the database contains all our server  \ instance .database information and flowers are blooming; things are good. If this doesn’t sound like anything you have done, head back to the Introduction to see if you missed something.

Now it’s time to get everything connected so we can just fire up a GUI and press some buttons, to get the data we need fast.

How the Sausage is Made

If you’ve been following along with this series, and you’ve set up everything as instructed, then you should be able to download the pre-made GUI script and run it out of the box. If you’re pointing to a custom instance or database just change the $RepositoryInstance and/or $RepositoryDB before firing it up. If you want to learn more about how this was put together, keep reading. If you don’t care how the whazits work, you’re done.

At the top of our list is to create a form with buttons and give them names so we can call them in our Powershell. You can either build the form manually with this guide here, or use Visual Studio*. I’m going to be using the latter method because it’s the most versatile, and frankly the easiest. If you use the former, then you’re kind of on your own. Sorry.

In the  Visual Studio method (I’ll be using 2013 Ultimate) you’ll be utilizing Windows Forms and then running them through a “cleaner” to make them Powershell ready. This guide at FoxDeploy explains the whole thing spectacularly and shows you how to create some very complicated UI’s that are Powershell-friendly. I’d recommend going through Parts I and II as they’ll be the things you’ll need to create what we’re going to use here and then come back. Don’t worry, I’ll wait.

Then We Build

Got the GUI code? Cool. The first part of Stephen’s code uses a -replace to filter the Windows Form code and make it work in Powershell. I took that piece and made it a second script so I could just have the clean version of the XAML in my final code. You can find that code here.

Just copy/paste your <Window>…</Window> code over the commented area and run the script. It will spit out the final code and tell you all of the objects you can tie actions to (Name, Value). Then drop in Stephen’s XAML reader code to the main script with the cleaned code and you should have a GUI… that does nothing.

Whee.
Whee.

As I mentioned before, when you pushed the XAML code through WPF_to_PSForm.ps1 it will tell you what the objects are on your form. For our purposes, this is simply a few buttons that need to be tied to stored procedures. We do this though .Add_Click() as in the example below:

$WPFBt_All_Data.Add_Click(
{
$sqlCommand = "
EXEC dbo.prGetInventory;
"
$dataset = Invoke-SQL -datasource $RepositoryInstance -database $RepositoryDB -sqlCommand $sqlCommand
Write-Host $dataset
$dataset | Out-GridView -Title "Database Inventory"
}
)

Nothing crazy that we haven’t been doing other than using Out-GridView. This cool little cmdlet pushes datasets out to a customizable table with filtering, sorting the ability to remove columns etc. -Title “SomeTitle” is the window title.

Sample sorted Database List with a few columns removed.
Sample sorted Database List with a few columns removed.

Once you’ve coded all of the buttons, then add the form display at the bottom, using out-null to suppress messages:

$Form.ShowDialog() | out-null

And that’s done. A Winner is You!

What now?

Using these scripts, you can go out and grab any information from the servers\instances you specify, pull it back into a centralized location and then use a GUI front-end to make fine tuning and retrieval easy. As I stated previously, this is a bare-bones system to centralize your database information. You can gather any piece of information from the Server, Instance or Database level by using the same tools that are currently collecting and retrieving this information.

It’s been a long journey, but thanks for sticking with it! If you want to make any alterations to the code or tighten it up (Odin knows that it needs it), feel free to make the changes and shoot them back to me. I’ll definitely give you credit for significant changes in this blog or the code itself.

Also, and I think this goes without saying, but if you want to use this in your personal or business environment: have at it! Just please make sure you give me proper credit, with maybe a link back to my blog/Twitter/Linkedin? That’d be super cool of you.

Thanks again and happy Inventorying!

–CJ Julius

*Full disclosure: I have not tried this with the Community version of Visual Studio, so all the features may not be there.

Creating a Simple Database Inventory Manager with Powershell – Part I: Building the Repository Database

I've got it. We'll put the databases in a database
I’ve got it! We’ll put the databases in a database.

This first part is simply setting up the database and the tables underneath of them. I’ve tried to make this as painless as possible by providing scripts do do most of this for you. I’ll use this infrastructure piece to explain some of the data that we’ll be pulling back.

This is by no means an exhaustive list of all the information that can be retrieved, but it serves as a foundation to show how all data could be pulled back to these tables. The only limit is determining how you’re going to retrieve this data.

This part will require some legwork from you as entering the ServerName and InstanceName with corresponding Id fields will be necessary. You should only have to do this once*.

If you’re unclear about what all this is then, and think you missed something, check out the Introduction.

[DBAdmin_QA]

[DBAdmin_QA.sql] – Setup script for 2012 can be retrieved here.

This is the database that will act as the repository for all of the information we want to collect. You can call it whatever you want, as you can change this in the code later, but I don’t recommend it the first time around. Why make it more complicated than it needs to be?

Make sure you take a look at the settings and verify they fit to your environment. It is intentionally small and grows slowly as we most likely will not be making it larger by leaps and bounds.

You will want to also make sure that the user you will be making updates/deletes/etc as, has full access to this database. This is a ‘durh’, but I always have to say it.

[dbo].[ServerList]

[dbo.ServerList.sql] – Setup script for 2012 can be retrieved here.

This is our master list of all Servers that contain SQL instances we want to know about. Initially, you’ll want to insert all the Server Names (fully qualified if necessary for your environment) leaving the other fields blank. I don’t put things in bold unless they’re important.

An example of T-SQL code to insert these items:

INSERT INTO dbo.ServerList
( ServerName )
VALUES
('SOMESERVER'),
('SOMEOTHERSERVER')

Notable Columns:

Id – Identity Column for all Servers
ServerName – Manually entered Server Name
IPAddress – IP Address of Server
OSName – Name of the Operating System
OSServicePack – Number of the Service Pack

[dbo].[InstanceList]

[dbo.InstanceList.sql] – Setup script for 2012 can be retrieved here.

Contains all of the Instances and the proper ServerListId (Foriegn Key to dbo.ServerList.Id) as well as some related information. You’ll need to INSERT all of the Instances and match them up to the ServerList.Id column when you inserted the servers.

At some point in the future I may add to the code where it dynamically builds the InstanceList, but that has not been done. This is because there are some instances that ‘exist’ on servers are shut down or disabled in some way. This allows the Instance to be listed in your Full Inventory even if it can’t be accessed. Wouldn’t be much of an Inventory if we weren’t able to inventory it.

An example of T-SQL code to insert these items:

INSERT INTO dbo.InstanceList
(InstanceName,ServerListId )
VALUES
('NAMEDINSTANCE',1)
,('MSSQLSERVER',2)

If you’re a lazy bum and don’t like matching ServerList.Id’s to InstanceList.ServerListId:

INSERT INTO dbo.InstanceList
( InstanceName
,ServerListId
)
SELECT 'NAMEDINSTANCE',
sl.Id
FROM dbo.ServerList sl
WHERE sl.ServerName = 'SOMESERVER'

UPDATE: I’ve made this even easier with a new stored procedure Utility.prInsertNewServerAndInstance. If this is your first time seeing this then there was no update and this procedure has always been here. Look! Over there! Something distracting!

Notable Columns:

InstanceName – Manually entered name of the Instance
ServerListId  – Manually entered FK to dbo.ServerList.Id
MSSQLVersion – Version of MSSQL running this Instance
MSSQLVersionLong – The long-form version of previous column
MSSQLServicePack – Current Service Pack of the MSSQL engine
MSSQLEdition – Edition of the MSSQL engine
isProduction – Manual bit flag to designate a production instance

[dbo].[ServiceList]

[dbo.ServiceList.sql] – Setup script for 2012 can be retrieved here.

The dbo.ServiceList table contains information on MSSQL services running on the server. That does include the MSSQL Database Engine, but also other items such as Reporting Services or Full Text Services.

dbo.ServiceList is dynamically filled so there is no need to add any information to this table.

Notable Columns:

ServiceDisplayName – The Display Name of the Service
ServiceName – The system name for the service
ServiceState – State of the Service (ie “Running”)
ServiceStartMode – Start Mode of Service (ie “Auto”)
ServiceStartName – User the Service is running as

[dbo].[DatabaseList]

[dbo.DatabaseList.sql] – Setup script for 2012 can be retrieved here.

This table is the dynamically generated list of all the tables in the database. As you’ll see later in the Powershell, anything we can pull back from sys.databases (or anything we can join to it) can be put in this table or a related table.

Notable Columns:

DatabaseName – Name of the database (crazy, right?)
SizeInMB – Size of the database returned in Megabytes

dbo.DatabaseList is dynamically filled so there is no need to add any information to this table.

Database Ready

And that’s it for the tables and database. Keep in mind that this is a basic structure that can be the core of any setup you’d like. Any data you can capture via T-SQL or Powershell can be compiled and put into these tables or better yet, other related tables.

In the next post, I’ll talk about how we’re going to be allowing this information to be put into the database via stored procedures.

 

*Unless you forgot to make backups and accidentally wipe the tables. In this case, you can think about your mistake while you re-enter all the data again.

–CJ Julius

A Collection of Collections of Free Microsoft Books

Image modified from a free one provided by http://www.norebbo.com/
Microsoft has lots of free stuff out there. Image provided by norebbo.

There are a lot of free materials out there for learning Microsoft products, and suprisingly (or not?) a lot of them are from Microsoft themselves. I thought I’d take a moment to organize and collect my list of free resources in the hopes that not only will it help me organize and find what I need, but also help others of you who don’t know about this stuff.

The one main source I’m using here is the MSDN MSsmallBiz  blog with posts by Eric Ligman. There are a massive number of titles to look at, but I’ve not seen them compiled into one place. Keep in mind that some of these are older and all the links may not work. I will update this list in the future if I find new/interesting free education materials in this genre.

The Collections

Huge Collection of 60+ MS titles on various topics

This was the first list to go up and start the whole series. Almost all of the offerings come in multiple formats (PDF, EPUB, MOBI).

Noteworthy sections:

Visual Studio 2010 – Office 365 – Windows 8 – SQL Server 2012


 

Large Collection of 20+ MS titles on various topics

The second in the series, and the least interesting of the groups, but it does come with some interesting titles. This group only comes in one format: PDF.

Noteworthy sections:

Own Your Space (a book for teens, no really) – SQL Server 2012 Dev Training Kit


 

Gigantic Collection of 200+ MS titles on various topics

The last group contains quite a few of the previous two sections (but not all, I’ve found). Most are in PDF or DOCX (word) format with a few in portable and non-portable formats thrown in.

Noteworthy sections:

MS Office – Powershell 4.0 (this stuff is really good) – CRM – Quick Start Guide group – even more SQL Server 2012

Bonus

If you’re looking for information on specific Microsoft technologies or if you’re gearing up for an MS cert, check out the Microsoft Virtual Academy.  They’ve got kind of a neat gamification thing going on where you get points for completing certain courses.

 

-CJ Julius

Syncing Between Linux and Windows with BitTorrent

Skip the insecure Cloud with BitTorrent Sync
Skip the insecure Cloud with BitTorrent Sync

I’ve always been a DIY kind of guy when it came to technology, and the idea of giving my data to cloud services such as Dropbox or Box.com (and whoever has access to that data besides them) seemed a little iffy. The cloud, as great as it is for some things, isn’t really built for too much security. Keeping data private on an internal system is hard enough, but throwing it out to the internet only multiplies these issues.

That’s where BitTorrent Sync comes in. Built by BitTorrent Labs (and using the BitTorrent Protocol), this solution boasts that it will allow you to sync between different OSes, securely, and without throwing any of it out to the cloud. This increases security incredibly, and isn’t that hard to set up. I put it on my Linux laptop (Stu) and a Windows 8 desktop (Zer0), both of which I’ve used in previous projects. It works, but it has a few caveats as you’ll see below.

Installation on Linux

Linux installation is fairly easy, if a bit obtuse. Instead of an installer of any kind, the package for BitTorrent Sync comes with a License.txt file and a single btsync binary. To start up the software, simply unpack it, navigate to the containing folder in a terminal and run the ./btsync command. That’s it.

[code]$ cd /Location/of/File
$ ./btsync[/code]

The Linux binary can be configured through the webGUI (kinda) or the more robust sync.conf file.
The Linux binary can be configured through the webGUI (kinda) or the more robust sync.conf file.

However, unlike it’s Windows and MacOS brethren, there’s no independent GUI to use. You’ll need to open a browser and head to a webpage to administer it. In most cases you can use the address 127.0.0.1:8888

From there you can select the folder you want to sync as well as generate a secret key for said location. The key is to allow other computers on your network to access the folder securely. Barring any conflicting firewall settings on your local machine, this should just be a matter of putting in the secret when you add a folder.

If you need the key from a folder you’ve set up previously, you can get it again from the gear icon next to the listing in BitTorrent Sync. Also, if you head to the Advanced tab you can grab a “Read-Only” secret. If you use this key when setting up another computer, it will read from the folder but never write to it. This is useful if you want the updates to go only one way or you want to give someone the ability to see what’s on your machine without running the risk of them deleting or altering the files.

Installation on Windows

Next, I went to Zer0, my Windows machine, and installed the software. From what I understand, the Windows and MacOS versions are pretty much the same, so other than the intricacies of the Mac platform the installation and use should be very similar.

The Windows application is a little plain, but gets the job done.
The Windows application is a little plain, but gets the job done.

After running the installer, you’ll be presented with a page that has several tabs. Go to the “Shared Folders” tab and click on “Add”. Put in the secret from the share that we want to access and click “Okay”. It should have all the information it needs to connect and start syncing. Mine did it automatically and pulled the four or so test files with no further work on my part.

You can also add a local folder and sync it here. By default it’s the btsync folder in your Documents directory. I just left this as it is for my testing purposes.

Tweaking the System

Now that it’s set up, you can do a few more things to shape it to your preferences. As you first may have noticed you can add any number of folders to sync, for no cost unlike most cloud services. So if your primary concern is just moving files back and forth behind the scenes (as I do) then that’s probably this setup’s greatest strength beyond security.

There are further options as well that fall into the more advanced users’ category. On the Preferences page in both the Linux WebGUI and the Windows application, you can set rate limits, alter whether the software loads at boot and some other odds and ends. In the Advanced section, you can do even more. Here’s a quick rundown of these options:

The conf file has pretty good explanations for every editable line
The conf file has pretty good explanations for every editable line

disk_low_priority: If True, BitTorrent Sync will set itself to Low Priority on the system. Turn this on if you’re noticing serious speed problems when using BitTorrent Sync

lan_encrypt_data: If True, BitTorrent Sync will encrypt data sent over the local network. Turn this on if you want to hide your traffic from others who may be using the same network as you.

lan_use_tcp: If True BitTorrent Sync will use TCP instead of UDP for local transfers. Will use more bandwidth but will be (at least theoretically) more reliable.

rate_limit_local_peers: If True, BitTorrent Sync will apply rate limits (set in General Preferences) to local users. By default rate limits are only applied to external peers (those not on your network).

In Linux, these options as well as a few others are all stored in the configuration of btsync. You’ll need to go to the folder that you have btsync running in to access it. First, you’ll probably want to output a sample configuration and open it in a text editor to see all options you have. There are quite a few.

[code]$ ./btsync –dump-sample-config > sync.conf
$ gedit sync.conf[/code]

It’s pretty self-explanatory, but I want to direct your attention to the username/password fields. Remember that webpage we went to earlier to set up the shared folder on Linux? Well it’s actually hosted from your machine, meaning that anyone who as the access to the network can pull up your BitTorrent Sync options and mess with them. So it might behoove you to set this option.

Once you’ve organized things the way you want them in your sync.conf file, save it. Now, you can import it back into the BitTorrent Sync application by running btsync with the modified conf file as such:

[code]$ ./btsync –config sync.conf[/code]

Worth the Effort?

And that’s pretty much the ins-and-outs of the BitTorrent Sync application. I imagine that I’ll be using this not as my primary software to sync things between machines or as backups, but I will have it move files and folders from one machine to another periodically. Perhaps one could set up a backup drive on a server that just copies one way from all the machines that are linked to it. I imagine that could be a project for a different day.

On the whole this is a nice piece of software that pretty much does what it says it’s going to do, and securely. I know it’s Linux, but the lack of a real GUI and the complication of editing advanced options by way of the .conf file is kind of a downer. I’m totally fine with using the command line (in some cases I prefer it), but that drags down the score a bit on this one because it’s not very user friendly. Still, a fine piece of software that I will definitely be utilizing in the future.

Rating: 4.5/5 – Pretty darn good. However, the Linux version takes a little work to get customized and the Windows/MacOS advanced pages are a little confusing at first.

-CJ Julius

Online Privacy: How to Get It and How to Keep It.

Is complete privacy online even possible?
Is complete privacy online even possible?

In the wake of the NSA/British Intelligence scandal, and the continuing surveillance of Internet Service Providers and websites such as Google, the interest in personal privacy has grown. While this article won’t be a long-form argument for personal privacy (mostly because I don’t think I need to do so), there are a few relatively easy things you can do to keep your online persona under your control and there’s good reason for it.

The oft-repeated adage is that “you shouldn’t put anything on the internet that you want to keep private.” While this sounds logical and simple, it usually isn’t. So much of what we do is tied up in the Internet. If you’ve ever bought anything online, done a web search or even paid a bill via a website, then that information is stored somewhere and is accessible to someone. And, while many make the argument that they have nothing to hide, the truth is that you probably do.

Not all of us have a murder or mob ties to cover up necessarily, but almost everyone has a debit/credit card information that we don’t want out there, or a few less-than-flattering pictures. On a different note, just because what you’re doing isn’t illegal, it doesn’t mean that you want to broadcast it to the world. In fact, you might be breaking the law without even knowing it.

That said, what do we do about it? Is there any way to hide everything we do on the net from everyone? The answer is: not really, but there are things you can do to minimize the amount of data you drop into the internet, and at best make it anonymous (not directly tied to you).

I’m going to outline a few steps you can take if you’re concerned about your privacy that will give you the most return for time invested. Much like my recent post on internet security, this is a short list of simple to do things that give you the greatest “bang for your buck”.

Your Browsing and Searching

The browser is where most websites will get the information they collect on you. Most of it is pretty general, the OS/browser you’re using, how long you were on the site, and things like that. However, sites that are more clandestine or that you use frequently can collect a large amount of information about you.

Firefox can be set to wipe everything but passwords every time it's closed.
Firefox can be set to wipe everything but passwords every time it’s closed.

Take Google for example. This is a website that we know collects data on its users and we know has been syphoned by the NSA (National Security Agency). When you log into any of their services, or do any searches from the site, all that information is stored and linked together. This data, over time, can build a pretty accurate picture of you based on your search and browsing habits. It’s not even necessary for you to give Google a name for them to find out who you are, as this can be mined from the data you give them. If you’re constantly going to a few sites and logging in, and any one of them has your name anywhere on it, then that can be linked back to your data.

The data doesn’t even necessarily have to come from you. The recent Facebook breach allowed people to access the contact lists of people they didn’t even know and download them. If you are in the contact lists of people who have Facebook and they’ve uploaded their contact lists to Facebook, then you’re on the site… even if you’re not on the site. Your information can be compromised if you’ve never had an account.

There’s not too much that you can do about the Facebook debacle, short of making sure that no one who has you as a contact uploads their data to the site. Though, there are a few things that you can do in general to reduce your footprint online.

As mentioned in my Internet security post, set your browser to hide you online. Most major browsers now have a “Do Not Track” option in them that will tell sites that you want to opt out of being watched. Most “good” sites will honour this and not track you. However, a few will still do so.

DuckDuckGo is not the most powerful search engine, but it's definitely the most stealthy.
DuckDuckGo is not the most powerful search engine, but it’s definitely the most stealthy.

To combat this, we need to take the browser work a bit further. Having the browser automatically use incognito mode (Chromium/Chrome) will greatly reduce the amount of tracking data that the browser can pass on. However, incognito mode can cause problems with certain websites, so, you can do like I do and have the browser clear everything every time you close it. Firefox has this option, and while it’s not as robust as the incognito/stealth mode, it does make browsing significantly easier. Every time I close the browser and reopen it, it’s like I’ve just installed the browser; websites have nothing to track because as far as they can see, I’ve never been to any websites.

Now, if you don’t want to make any changes to your browser or you want another layer of security, you can change the search engine that you use. While the biggest ones such as Yahoo! and Bing also collect your data and share it, there are ones that are built specifically with privacy in mind. The main engine I use to do all of my searching is DuckDuckGo which keeps no logs on its users and sets up an encrypted connection (via SSL) between you and the search engine so nothing can be intercepted.*

Using the above techniques you can keep your search history private, or at the very least separate you from your searches.

Your Connection and Software

This is all well and good, but it doesn’t protect you against someone snooping on your connection to the internet. Even though you’re anonymous to the search engine, you’re not so anonymous someone who’s watching you browse, such as your ISP or someone sniffing packets in a cafe. To secure that, we’re going to need to hide your internet connection.

The easiest (cheapest) way to do this is to always try the https:// version of a website before the http:// (note the “s” for secure). This little change will create a secure connection between you and the website, making your traffic unintelligible to a malicious viewer. Not all sites support this, but some of the big ones do. The site you’re going to will still be visible, but the contents will not. Keep in mind that this is the “free” option and is very hit-or-miss.


Private Internet Access’s “Why use a VPN?” video.

Another option, which is the route I would recommend, is to push all your data through an encrypted VPN (Virtual Private Network). There are a lot of them out there, depending on how much privacy you want and what price you’re willing to pay for it. Some offer a full range of services including news access as well as other benefits like VyperVPN (will run you about $20/month) or simple unlogged access like PrivateInternetAccess **(about $4/month). In both cases, the system creates an SSL (Secure Socket Layer) VPN between you and their servers and then pipes you with an anonymous IP out to the internet.

Someone spying on you would only see a mass of garbled data being sent to some server somewhere where it disappears. Any website or person on the internet would see your data coming from a block of IPs owned by a VPN company. There’s virtually no way to connect the two (no pun intended).

Skype will allow you to stop cookies and not keep a history as well as other privacy options.
Skype will allow you to stop cookies and not keep a history as well as other privacy options.

If you’re only concerned about eaves-dropping when you’re out and about, you can also use something like Hamachi Log Me In to create an SSL VPN between a mobile device/another computer and a home machine. Keep in mind that with this system, anyone watching your home machine will be able to see the data unencrypted. The secure connection is only between your remote device and the home computer.

Lastly, the software you use on the internet that isn’t your browser, such as Skype or Yahoo! messenger is also targetable. While there’s only a little you can do to secure these, you can do a few things. First of all, check your privacy settings and make sure you have everything locked down. Most of these services have a small but useful section in the options called “Privacy”. Also, make sure your chat history isn’t being saved. You can turn this off in every messenger. While it doesn’t guarantee that the data isn’t being stored elsewhere, it does reduce the lifetime of the data and the chance that it will be recovered.

Am I Private Yet?

So the question remains as to what affect will all this have? The truth is that we don’t know entirely. Depending on who’s targeting you and why, the things listed above, if implemented properly, can range from significant annoyance to complete blackout. However, if you implement no privacy measures you can rest assured that some, if not all, of your data is being collected and catalogued.

Tor is a more advanced way of getting privacy online, but it has it's own weaknesses. Check it out here.
Tor is a more advanced way of getting privacy online, but it has its own weaknesses.

Not all of these may be for you. But a smattering of them in some form or another will help, especially the VPN services, and I recommend you at the very least lock down your browser as mentioned above and in my previous internet security post. Even if you think you have nothing to hide, you may find out in the worst way possible, that yes, you did.

* You can get the add-on/plugin for your browser of choice as well so it’s automatically in the upper right search box on your browser.

** If you’re just looking for privacy and nothing else, this is the way to go.

The Easiest Way to Root a Galaxy Note 10.1

Unlocking your Samsung Galaxy Note 10.1 (Proceed with Caution)
Proceed with Caution

Rooting your device is a pretty dangerous game, even nowadays when it’s almost commonplace. You can lose your data, void your warranty or even brick your device. However, if you’re looking to really unlock the power of your Android tablet, there’s really no better way. It gives you unparalleled access to your files, ability to install apps that do some pretty amazing things, or even install your own OS aside from Android.

So, obvious warnings aside (above), I want to show you the easiest way I’ve found to root the Galaxy Note 10.1, my favorite tablet. Ideally, you’ll lose no information and really not notice much of a change to the OS, excluding a new app that manages root access. It’s also really quick (maybe 30 minutes at most).

However, one last time, I am pointing out that this can destroy everything, and is just here for your edification; I take no responsibility for you turning your Galaxy Note into a dinner tray.

The Tools

There are four pieces of software and two pieces of hardware that you’ll need for this.

Software:

  1. Windows – Sorry, but you’ll need to use Windows Vista or later (I haven’t tried on XP, so I have no idea, but I imagine it will work)
  2. Android 2.2 or later – Works all the way up to Jellybean 4.1 that I’ve tested.
  3. Samsung Kies – This is the software that you’ll need to set up your computer to modify the system files we’re going to be changing. [Download]
  4. CF-AutoRoot – The rooting software itself, just open this link in a tab and we’ll get back to what you need later. [Download]

Hardware:

  1. Samsung Galaxy Note 10.1 – Your tablet, durh.
  2. Charging Cable – Has a 30-pin connector on one end and a USB on the other.

The Process

First of all, install Samsung Kies if you haven’t already. At the end of the install select the option that you don’t want to launch it (there’s no reason to, we’re not going to be directly using it) and to launch in Normal Mode. That’s all we’re doing with that.

This is what it should look like if you've done it right.
This is what it should look like if you’ve done it right.

On your tablet, open Settings and go to the “About Device” page, usually located at the bottom of the options. You can see your Android version here (make sure it falls in the range listed in tools) and the Model Number. It should be something like GT-N8xxx. Remember this model number.

Look at the page for the CF-AutoRoot forum that is listed above and find a link with your model number in the third post. It must EXACTLY match that number. Download that zip file and unpack it somewhere.

Now, on your tablet, hold in the Power Button and the Volume Down button for about 10 seconds. Your tablet will take a screenshot and then reboot. Keep holding down the buttons until you see a diagnostic screen with an Android symbol and the WARNING page which you should probably read. We’re reaching the point of no return. Click the Volume Up button to acknowledge that you understand the risks. You will see a “Downloading…” message.

This new app will be installed after reboot and will let you manage how apps get access to root.
This new app will be installed after reboot.

Plug the charging cable into the tablet and into a suitable USB port on your computer. Set the tablet aside and leave it alone.

Open the Odin-vxxxxx.exe as an administrator.* You should see a yellow box with COM1 or something like that in it. If not, reboot your computer and try again, starting from the beginning of this paragraph.

In Odin, click on the PDA button. Navigate to the folder where you unpacked Odin executable, select the .MD5 file (it should be the only option) and open it. Click the Start button.

After a few seconds the first box should turn green and say PASS. Once it does the tablet will reboot and your tablet is now rooted.

The Aftermath

Root Explorer App is a really powerful tool, especially for the price (free).
Root Explorer App is a really powerful tool, especially for the price (free).

This “autoroot” is the simplest of roots and keeps your tablet as close to stock (as Samsung released it) as possible. It gives you and apps superuser access and manages them through a new app called SuperSU. This is perfect if you’re looking to install some stuff that needs Superuser access or you want to go poking around yourself. However, this isn’t tailored for power users or those who want to supplant Android with another OS.

The first super-user required app that I send most people to is Root Browser on the Play Store. It’s a powerful file browser and it’s free. Be careful not to damage your system!

*If you don’t see the Odin executable, then you may need to get it separately. A stand-alone version of Odin can be found on the same forum attached to the second post. Just download it and extract it to the same folder as the MD5 and continue on.

-CJ Julius

How I Got My Android Tablet to Boot Windows 95

I was rummaging through some old software of mine a few weeks ago and taking stock of the old operating systems that I had commercially. I noticed that along with some older versions of Redhat and Ubuntu Server, I owned every version of Windows since 95, including quite a few server versions. I wondered what I could possibly do with them, since I don’t even use my store-bought copy of Windows XP anymore.

Hey, I remember you.
Hey, I remember you.

Then I looked at my new Galaxy Note 10.1 tablet and got an idea. I wondered if I could get Windows 95 to boot on it. So, I fired up Virtual Box and an old machine I had and got to work.

Note: I am using Ubuntu 12.04LTS and a Galaxy Note 10.1 to do this project. Also, I had access to another, older machine with which I could install Windows 95 myself. Your mileage may vary.

Build 95

There are a few ways to go about this. One is to use Virtual Box to create working Windows 95 VDI file and then convert that to an IMG after you’ve got it running and another is to just find a computer with Windows 95 and make an image of the drive. Either way you’ll have to do three things:

  1. Install DOS 5.x or better before installing Windows.
  2. Install Windows 95 and get it working.
  3. Make your image (.IMG) file.
Click to Enlarge
In Virtual Box, you’ll need to set up an MS-DOS environment first and then probably migrate to 95 later.

Now, I’ve tried both ways, and they’re both complex. In the first example, using Virtual Box to create a Windows 95 compatible area for the OS to work in is a pain. This is because the Windows 95 disk is not bootable (and neither is Windows 98 for that matter). You have to have DOS 5.x or later installed first and THEN go to Windows 95. This is as much work today as it was back in when Win95 came out.

Then, once you have Windows 95 running you need to get all the drivers (and you’ll probably have to use an older version of Virtual Box because of compatibility issues), some of them custom-made, install them, and squash bugs as they come up.

When you have everything set up Virtual-Box side, you can convert the VDI to an IMG file to make it usable with the vboxmanage command in termninal:

vboxmanage clonehd Win95.vdi Win95.img --format RAW

This is not the method I recommend, as it is the hardest even with a walk-through, however it may be the easiest for people with limited access to hardware. I had, luckily, a piece of hardware that would run Win95 with minimal effort so I went that route.

First, I put I installed MS-DOS 5.0.7 (available legally and for free here) from some image files to actual real-live 720KB disks. Yes, I still have a few of those. Then I set up my CD-ROM*, no small feat, and began the Windows 95 install.

Once this had been done, I pulled the HDD out of the computer and connected it to an IDE slot in another machine. I then used the dd command to make a raw image file of the newly-added drive. This ended up giving me a large file because I had given a Gig of space to the virtual drive so I’d have lots of space to move around. You could probably get away with only 200 or 300 MB if you wanted to do so. In any case, the command to image the drive was:

dd if=/where/drive/is/mounted/ of=where/you/want/image/ bs=4K

Now I had my Windows 95 image and it was time to get it running on the tablet!

Install 95

There are multiple ways to get Windows to run on your tablet once you have an image you like. I personally went through my version and pulled out all the things I didn’t want so I could create a smaller image. I eventually got the entire thing down to 200MB, but that was with a lot of work. There are also two ways to get the image running on your tablet. There’s the way I did it initially, and then the easy way. I’ll be showing the easy way and then give a brief overview of the more difficult path.

The Easy Way

You’re also going to want to use something like AirDroid, which I’ve reviewed before, to move the files over because chances are you’re going to be doing this a lot. As you make tweaks or move different things back and forth that GUI is going to come in real handy.

Click to Enlarge
After you put in the image location and name, it will need to copy it to the SDLlib’s directory, probably on your internal memory.

Move your image file over to your device and take note of its location. You’ll probably want to write it down or something, make sure you note the CASE of the letters, because that will be very important. Also you’ll need to make sure you have enough space to copy the image over to the working directory of the emulator that we’re going to use here in a minute. So you’ll need at least twice the space of the original IMG file to use it.

Go to the Play Store and find Motioncoding’s Emulator. It looks like an Android with the Windows XP flag colors on it. Download, install and run it.

Once running, go through the menus (using the forward/back buttons, it really couldn’t be more simple) until it asks you to install libSDL and do so. Then select the option under “Import from Library” to Add Custom Images. Name the image whatever you want and put in the path to the image in there. For example, mine is:

/storage/extSdCard/SDL/Win95.img

Select the image from My Images and continue to the end. You should see your OS boot.

The Hard Way

The reason I’m putting the hard way on here is because it gives you a bit more control over your install and, I think at least, runs a bit faster. In any case I’m going to assume that you’re doing it this way because you’re a little more experienced/curious and don’t need me to hold your hand.

Click to Enlarge
Copying over the SDL apk and related software.

Step one is getting a working version of the SDL apk and installing it. You can do a quick Google search for it, but I’m not sure of the legal ramifications (or its copyright) so I’m not putting a direct link here. Keep in mind that you will need to allow “Apps from Unknown Sources” to be installed on your device. This can usually be found in the “Application Settings” area, depending on your version of Android.

Place your Win95 image in the SDL folder with the APK and rename it c.img, and load SDLlib. You may have to do more tweaking at this point as Networking didn’t work out-of-the-box for me. I needed to modify some already existing .bin and .inf files to coax them into doing what I needed to do, and even then it’s a little haphazard. You’ll need to have some method of editing the img file if you can’t get networking going or you’re going to need to re-image the drive every time you want to make a change.

This way you’ll also have access to the BIOS and VGABIOS bin files, if needed, but I didn’t end up touching them.

Android 95

My reasons for doing this were purely academic. I just wanted to see if I could get it to boot and get it usable. After several weeks of poking at it I was, by all of the above methods, able to get 95 and 98 going this way. Windows 98 was just a matter of upgrading 95 and creating a new image file. I can’t think of many reasons to do this other than for the learning experience, though there are lots of pieces of software out there that don’t work so well in modern versions of Windows and maybe you want to take them with you.

Click to Enlarge
Windows 95 successfully running on my Galaxy Note 10.1 with mouse and keyboard support

Also, I was able to get my Logitech keyboard/mouse combo to work through the 30-pin charging port, and while dragging the cursor across the screen and “clicking” by tap was interesting, the keyboard is the way to go. It’s just too cumbersome for daily use otherwise.

So there it is, an Android tablet booting Windows 95/98! You can supposedly do this with Windows 2000 or XP, but I have not tried. If you have let me know, because I’d be interested in how you got native NTFS to work.

*There’s no instruction here because it really depends on your CD-ROM as to how you’d go about this. You’ll have to find one that will work with Win95 and DOS. I had one in the machine already so it was just a matter of setting it up manually through DOS.

-CJ Julius