EASILY Bulk Load files into a table with different columns

with NO FORMAT FILE!

I consider this a gem just hiding on the MSDN docs :

Using BULK IMPORT on a View

The following example creates the v_myTestSkipCol on the myTestSkipCol table. This view skips the second table column, Col2. The example then uses BULK INSERT to import the myTestSkipCol2.dat data file into this view.

In the SQL Server Management Studio Query Editor, execute the following code:

CREATE VIEW v_myTestSkipCol AS
SELECT Col1,Col3
FROM myTestSkipCol;
GO

USE AdventureWorks2008R2;
GO
BULK INSERT v_myTestSkipCol
FROM ‘C:\myTestSkipCol2.dat’
WITH (FORMATFILE=’C:\myTestSkipCol2.xml’);
GO

Say what? You can just write a view and not deal with a nasty format file? This is useful any time you have an additional column you want on your staging table, but it’s not in the file you’re importing, and want to add it as part of your processing. Either its a flag on where the file came from, the file name, the date loaded etc…

Lets pretend, as part of a larger ETL process I just want to import names of people that belong to different teams.

  • I get a file daily for each team with new members
  • I want to transform the data for all teams as one operation after bulk-loading each team file (preferably into the same staging table)
  • Each file has the same name, I get it from an API based on the team name so they look no different when downloaded
-- person staging table
Create Table StagePerson(
    first varchar(50) NULL,
    first varchar(50) NULL,
    team varchar(50) NULL
)

-- a view to match whats in the file
CREATE View vImportStagePerson as
 select first, last
 from person
go

-- my ETL process will download each file, then bulk load into the table
-- ETL loop: for each team
BULK INSERT vImportStagePerson
FROM 'C:\[team]\members.csv'

UPDATE StagePerson
Set team = [team]
where team is null

-- end for each team

-- do your merge of all new team members from your staging table WITH team added

This way I save additional hops just to add that pesky team name to my staging table. And your final operation can take advantage of a larger set per-transaction.

Posted in Uncategorized | Leave a comment

Get Value from XMLA result file in NAnt with xmlpeek

If I got paid for every hour spent wrangling with XML…

Oh wait, I do

Problem:

  • Read a scalar-result out of an XMLA response file from an MDX query run against a cube.
  • Store it in a NAnt property.

Jiest of my XMLA file:

<return xmlns="urn:schemas-microsoft-com:xml-analysis">
  <root xmlns="urn:schemas-microsoft-com:xml-analysis:mddataset" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<CellData xmlns="urn:schemas-microsoft-com:xml-analysis:mddataset">
      <Cell CellOrdinal="0">
        <Value xsi:type="xsd:double" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">1.09</Value>
        <FmtValue>1.09</FmtValue>
      </Cell>
    </CellData>
  </root>


Surprisingly I’ve had good luck parsing and reading XMLA files from Java, so I figured this would be a quick solution using NAnt’s xmlpeek.  Taking a look I can just create an XPath statement of “/return/root/CellData/Cell/FmtValue”. Guess again!  After much banging my head I found a cheating solution using the XPath: “//*[local-name()='FmtValue']/text()” which just says “get me the local node named FmtValue anywhere in the xml doc”.  Not pretty, plus it worked on every online tester I found, but still returned nothing in NAnt – even though it says “1 node found matching…”.  I had popped a couple namespaces in to the xmlpeek task, but I still don’t really understand them…

Somehow this xml/xpath evaluator makes it all very clear (toggle the “XPath” option), it gives a handy Namespace Reference:

Namespace Reference
Prefix Namespace
a urn:schemas-microsoft-com:xml-analysis
b urn:schemas-microsoft-com:xml-analysis:mddataset
xsi http://www.w3.org/2001/XMLSchema-instance
xs http://www.w3.org/2001/XMLSchema
c http://schemas.microsoft.com/analysisservices/2003/engine

a,b,c??  I don’t know either, but if it doesn’t have an obvious prefix I guess you just make it up?  At any rate, I added the above namespaces verbaetum to my xmlpeek namespaces, then I was able to construct a “proper” XPath to my liking: “/a:return/b:root/b:CellData/b:Cell/b:Value/text()”.  With NAnt it does the text() evaluation on it’s own so you only need “/a:return/b:root/b:CellData/b:Cell/b:Value”.  With that unlocked, you should be able to peek and poke any item within an XMLA statement spit out from Analysis Services.


	<xmlpeek
	file = "result.xmla"
	xpath = "/a:return/b:root/b:CellData/b:Cell/b:Value"
	property="your.property"
	verbose="true" >
	<namespaces>
		<namespace prefix="a" uri="urn:schemas-microsoft-com:xml-analysis" />
		<namespace  prefix="b" uri="urn:schemas-microsoft-com:xml-analysis:mddataset" />
		<namespace prefix="xsi" uri="http://www.w3.org/2001/XMLSchema-instance"  />
		<namespace prefix="xsd" uri="http://www.w3.org/2001/XMLSchema" />
		<namespace prefix="c" uri="http://schemas.microsoft.com/analysisservices/2003/engine" />
	</namespaces>
</xmlpeek>

Posted in Code, Note To Self, Work | Leave a comment

“@crashpaddesign: This is looking to be a serious contender to Shopify: http://t.co/iKL7oRh9 #woocommerce #ecommerce @Pirnke

“@crashpaddesign: This is looking to be a serious contender to Shopify: http://t.co/iKL7oRh9 #woocommerce #ecommerce @Pirnke
View on Twitter

Not everything I say on Twitter is important, but I thought this was. You should follow me @thegarrettp

Posted in Uncategorized | Leave a comment

Sticky/Squealing Ratchet Starter Briggs and Stratton

It’s taken me far longer than it should, but I’ve finally gotten my McLane Edger craigslist score in a proper working order. The original issue had always been an awful squealing noise that would totally un-wind the pull cord in disastrous fashion. A secondary issue was that the cutter arm was totally frozen, would not turn, rotate or telescope. I thought maybe the cutter arm was the issue as it seemed to happen as soon as I put pressure on the engine pulley. Turns out they were un-related, after failed attemps to free the cutter arm I ended up just replacing it with a new one – best $75 spent.

I knew then it had something to do with the starter pulley, which is a hard thing to google if you have no idea what it’s called. Common official terms appear to be “Ratchet Starter”, or “Ratchet Clutch”. What also makes it hard are the specs for this engine; a 2hp Briggs and Stratton horizontal shaft 4-stroke, I think they haven’t made them since the mid 90′s but they are bulletproof little engines used in everything from snow blowers to a favorite for go-karters.  If I knew something about engines, probably wouldn’t have needed to google anything…

The best resources I found we’re Canadians or upper mid-west folk’s youtube videos on repairing their snow blowers/reel mowers, this was by far at the top:
http://www.youtube.com/watch?v=S2Kn6RiTA7U

And this guy has nothing but time to dink with small engines:
http://www.youtube.com/user/pimpinpenz#p/u/39/Fz7RUU_-YSc

To repair:

Edger with pull starter housing on, you'll pull this off to get to the ratchet starter.

This is also an example of how to change the oil, plus stuff on blocks looks cool no matter what it is.

 

 

 

 

 

Ratchet starter assembly after removing pull-cord housing

This picture shows the ratchet assembly in the center after removing the pull-cord housing.

 

 

 

 

Inside of ratchet starter

This shows the inside of the ratchet starter.  I’d actually been to this point about three times, oiled and cleaned all of the ball bearings with no luck solving my problem.  Wasn’t until I watched that youtube video I realized that inner shaft was the problem.  I took that off, was able to clean the entire inside of the ball-bearing housing, and cleaned and lubricated the engine shaft and ratchet-post thing.

Now it runs like a champ, along with my new cutter arm it’s like new.  And yes, my yard looks better than yours.

 

Posted in DIY, Home | Tagged , , | Leave a comment

Install OTA HD Antenna – cut the Comcast Cord

I won’t rant for too long about the great Sham that is Comcast, but remember the big HD TV transition?  With Comcast you “didn’t need to do anything”, right – nothing except get some set top boxes for each TV in your home if you wanted to retain the same programming.  Oh – and you only get one for free with most cable plans.

Long story short, I found out with this set top box I didn’t get HD channels, yet my other tv with no set top box did get HD channels.  THE COMCAST BOXES BLOCK HD CHANNELS BY DEFAULT!  Why?  So they can up-charge you $$$ to upgrade, essentially, pay extra money to remove a filter on their boxes.

As an irate customer, I bought an HD antenna and let it sit in my garage for about 18 months, then I decided to install it:

Installed HD Antenna

Parts:

The Antenna I bought was cheap, outdoor, and had a nice swivel mount for direct mounting to the side of a house, on top of something, or pole mounting.  Turns out 1 1/2″ EMT conduit is really cheap, light, and sturdy.  I decided to go big and get it just above my power service mast.  Location was chosen based on the existing cable entering the house at that location, easy to mount by getting onto the garage roof, and it will be with all the other tall stuff attached to my house.  As soon as you try to home run your own cable more than once, you’ll desire good tools, so you can be slick like the Comcast man.  That Paladin set has been worth every penny, because did you know you can use Coax for signal cable too? Now you’re a home theater master.

Home entry point with Comcast service line

 

In a passing conversation about HD Antennas, someone at work clued me into the fact I want to make sure I ground it properly.  Something I figured the coax cable would take care of, but once you bolt a 8′ pole to your house that is now the highest point of your roof, its a good, easy idea.  I just bought a length of 14AWG bare copper wire that runs from the top of the pole with the cable to the house, then takes the most direct route to my existing home electrical ground conveniently placed near where my cable enters the house.

The Comcast cable already had a single ground block for the coax, which had a ground wire also connected to the home ground.  This was an easy swap out for a dual ground block.  From there the cable enters the garage to where it gets Amped and distributed through the house.

Cable entering the garageComcast also installed a signal amp to help with my internet and TV signal.  This is also helpful and usually recommended for HD Antenna installs.  So now in the garage, I have a Comcast line, and my Antenna line coming in the same spot.  This makes it really easy to plug and play between the two for troubleshooting, and whenever else in the future.

 

The results were OK, most channels come in fine, however a few don’t come in well at all. I have a feeling some alignment is probably necessary, that and getting my wife sold on the nuances of OTA cable.  Overall, its a success and if I had more time id figure out the alignment, or buy a better antenna.  Also, http://www.antennaweb.org/ is actually an awesome resource with the information you’d need and want about OTA Stations, strengths, and locations in order to determine optimal antenna, placement, and alignment.

 

Flying high and proud, like a symbol of freedom from Comcast

Posted in DIY, Home | Tagged , , , | Leave a comment

Get current domain from NAnt

I use NAnt for a build and deployment tool for databases and OLAP. I wanted to be able to add only the accounts needed per-environment to certain objects on the fly through the script, as they are run on whatever workstation they are being run from.

I’m using the “Primary Dns Suffix” of the machine – which for me indicates the domain we’re running in. (Dev/Test/Prod). I’m accessing this using ipconfig /all, saving the output to text, getting the value I want as a property for use later in the script.

<exec program="ipconfig" output="ipconfig.all.txt">
    <arg value="/all" />
</exec>
<loadfile file="ipconfig.all.txt" property="ipconfig.all" />
<delete file="ipconfig.all.txt" />
<property name="envrionment" value="dev" />
<regex pattern="(?<=Primary Dns Suffix\s(\s\.){1,7}\s\:\s)(?<environment>\w{1,})" input="${ipconfig.all}"/>
<echo message="CURRENT ENvIrONmENT: ${environment}" />

Regex Master.

Posted in Code, Work | Tagged , | Leave a comment

Analysis Services Add role members programmatically in C#

First, I don’t know why; but it seems like as soon as you add in the word OLAP or Analysis Services to your problem things just stop working as you would normally expect.

Problem:

  • I’m using NAnt as a deployment tool for Database versioning and MS OLAP deployment in multiple environments.
  • My service accounts aren’t standardized across environments (don’t get me started with THAT one…)
  • I want to apply different accounts, and ONLY the accounts for those environments to my cube roles.

The “should work out of the box solution”:


Server server = new Server();
server.Connect("mah-olap-server");
Database db = server.Databases.FindByName("MahOLAPDB");
Role role = db.Roles.GetByName("Role");

RoleMember r= new RoleMember("Domain\\Account");
role.Members.Add(r);
role.Update();

What’s the big deal right? Well, there are a few problems:

  • Adding it this way un-like through the GUI does not map the user account to an SID
  • Even after you’ve mapped it to an SID it will still add a duplicate role member.

First:

Server server = new Server();
server.Connect("mah-olap-server");
Database db = server.Databases.FindByName("MahOLAPDB");
Role role = db.Roles.GetByName("Role");
RoleMember r= new RoleMember("Domain\\Account");
if(!role.Members.Contains(r)){
    role.Members.Add(r);
    role.Update();
}

That is hopeless, I thought at first because of the SID thing – but NO… After the next solution, Role.Members.Contains() does not work very well even though in debugger I have the EXACT same role member I’m adding, and exists in the role. Not only was it not finding it, but it was still adding a duplicate. This along with the not mapping to proper SID – I deemed to be a future-issue I should avoid now. So…

List members = new List(); // I fill this with all the members I need to add
Server server = new Server();
server.Connect("mah-olap-server");
Database db = server.Databases.FindByName("MahOLAPDB");
Role role = db.Roles.GetByName("Role");
//oh wtf, I can't seem to Linq to the role.Members so I'll do it this way, plus case insensitivity on .Remove() doesn't exist
foreach(RoleMember rm in role.Members){
   string er = (from m in members
   where m.Equals(rm.Name, StringComparison.InvariantCultureIgnoreCase)
   select m).FirstOrDefault();

   if(er!=null){
       members.Remove(er);
    }
}

foreach(string m in members){
    //you'll want to wrap in try catch in case the account doesn't exist, you'll get a IdentityNotMappedException
    NTAccount acct = new NTAccount(m);
    //conver it to SID - check if its even real
    SecurityIdentifier sid =   (SecurityIdentifier)acct.Translate(typeof(SecurityIdentifier));
    //convert it back from SID  - this will get us proper account Name format/casing
    acct = (NTAccount)sid.Translate(typeof(NTAccount));
    //create my wholesome role
    //already done my "contains" magic, add that monkey
    role.Members.Add(r);
    role.Update();
    }
}

That final solution actually works out pretty well, and accomplishes a lot of items I would have skipped had the first one just “worked”:

  • Does the mapping of user accounts prior to adding to the role.  Make sure name format, and SID is added.
  • Creates the same behavior as when you add it through VS or SSMS GUI (if you add one that exists it does nothing)
  • Wasted countless hours
  • No over-engineered minute step of my deployment

And it did solve several problems along the way:

  • Creating RoleMembers with just the name does not map it to AD or a proper SID
  • Using the Role.Members.Contains() doesn’t seem to work well
  • You can’t make .Remove() on a List<string> case insensitive

I will try to write a follow up soon regarding how to determine what environment (primary domain your machine is connecting to) you’re in from NAnt

Posted in Code, Work | Tagged , , , | 1 Comment

ifttt.com is really slick 3 http://t.co/3ha12Wpr, testing some social integration

Not everything I say on Twitter is important, but I thought this was. You should follow me @thegarrettp ifttt.com is really slick 3 http://t.co/3ha12Wpr, testing some social integration #wp

Posted in Uncategorized | Leave a comment

SimpleDB: non printable charachters

“this is my sweet string” == “dGhpcyBpcyBteSBzd2VldCBzdHJpbmcZ”?

Those two are actually equivalent, as proven by Amazon SimpleDB.  We started seeing these mysterious strings in our SimpleDB data which is supposed to be  a direct upload of SQL data for use in a UI.  I automatically assumed it had something to do with special characters and proper encoding, as we have seen in our processes before.  But this case was more unique because instead of just mangling the special character, it has managed to blow out the entire string… WTF

The culprit was an “End of medium” ASCII control charachter.  These ASCII control charachters are all non printable.  Once I had this figured out some more googling led me to the answer of why the whole string was unrecognizeable, base64 encoding:

What’s happening/changed: http://www.dibonafide.com/?p=25

The official documentation: http://docs.amazonwebservices.com/AmazonSimpleDB/latest/DeveloperGuide/index.html?InvalidCharacters.html

In my case the charachters were more of an anomaly and I just wanted to be rid of them.  I had some options of taking care of it during the upload to SimpleDB or at the DB level.  What worked easier was just throwing this handy function on my DB, and converting in the view I’m using to get data to upload:

http://iso30-sql.blogspot.com/2010/10/remove-non-printable-unicode-characters.html

Per the official documentation above the response object for the item will actually indicate what type of encoding it’s using.  A few sites have mentioned they just base64 encoded on upload, and then decoded when its being used for display.  I think with it specified in the item itself, you can probably just design your UI or whatever is consuming the data to check for base64 encoded strings, then decode, and remove any invalid characters there.

Now if I could just get WordPress to stop using invalid XML characters in its markup… that’s a topic for another day!

Posted in Work | Tagged , , , , , | Leave a comment

Usability: Degraded

Recently I’ve come across two cases where it appears in the interest of usability, we’ve gone too far and invaded what I thought used to be “boundaries” for UX, and proper accessibility practice.

1. Google Hijacking My Down Arrow

Have you ever gone to a google search result page and tried to use your down arrow on the keyboard to bump down your scroll?  AND NOTHING HAPPENS?  They’ve hijacked your down arrow and given you what’s been affectionately named (by the internets) “little blue arrow” which is a cursor for each search result.  Each press of your down arrow will move this cursor to the next result.  Before: One keypress of down arrow would show roughly 10-20% more page “below the fold”. Now: I have to press my down arrow as many times as there are results showing “above the fold” to see more of the page…

I don’t know if this is some kind of conspiracy to increase views of top ranking search terms or just a bad attempt to “improve” usability, I’d rather it be the former in my book.

http://www.google.com/#sclient=psy&hl=en&source=hp&q=google+hijacking+down+arrow&pbx=1&oq=google+hijacking+down+arrow&aq=f&aqi=q-n1&aql=&gs_sm=e&gs_upl=2001l6244l0l6343l30l18l1l4l4l0l267l1988l4.5.4l13l0&bav=on.2,or.r_gc.r_pw.&fp=996a5a2e91f3c54d&biw=1086&bih=723

2. Sites are copy-jacking me

Yes it appears the current definition of “copyjacking” has something to do with copyright stuff, but that’s stupid.  So I’m on a news article, and I copy a line of text:

The vote had no legal bearing on the state’s mega-project

I paste it in order to tweet something and this is what I get:

The vote had no legal bearing on the state’s mega-project
Read more: http://www.seattlepi.com/local/transportation/article/On-Ref-1-Seattle-says-build-the-tunnel-2076294.php#ixzz1VIzuHvAE

Not just text that I didn’t copy, but its got two whole line breaks, blowing up my little tweet window…  Even worse than Seattle City Council wasting our time on votes and referendums that have “no legal bearing”

Based on my scientific Google research the practice seems fairly new, and the culprit on the site I was viewing was Tynt, as noted in the post below:

http://stackoverflow.com/questions/1203082/injecting-text-when-content-is-copied-from-web-page

I haven’t had time to crack open the js, but if they were also tracking what I copied it would be a great opportunity for a little #writealetter hacking to tell them what I think.  Maybe this atrocity will warrant general analytics-spamming, we’ll see who else is on my list.

Posted in Code, Rant | Tagged , , , , , | Leave a comment