BizTalk Services Announced

Some time back when I blogged about the future of BizTalk I got some pretty interesting comments , at that time I felt I could not reply more as it was still under NDA . Now that it is not as of today Microsoft is announcing BizTalk in the cloud or to give the newly christened name BizTalk services

What are BizTalk Services?
These services, which have been in internal incubation for the past year, represent hosted versions of some technologies developed in the Connected Systems Division. Included in this set of services are:
o Message routing – think of this as firewall friendly B2B messaging (Available now)
o Simple publish/subscribe event brokering – Pub/Sub at Internet scale (Coming soon)
o Simple federated identity and access control (Available now)
o Workflow processes – Simple templates for cross-organization integration and the orchestration of business processes interacting with multiple services (Coming soon)

Now that these services are in external mode ( pre-alpha ) MS will now host and manage the Infrastructure on a best-efforts basis. (No SLA) . This is presently free though need not be down the road .

This move as you will probably realise is in essence with Microsofts Software + Services vision

BizTalk Services will be available via labs.biztalk.net.

Morfik Patents AJAX compiler

Just saw this post at slashdot . From all I understood about reading through the whole text , It converts code written in any language to another format( Let us ignore that it is Javascript for a moment here ) and the fact that it is now becomes executable in browser . Doesn't that make it a compiler of some sort . and here I was thinking you cant patent something like a compiler unless you are essentialy demonstrating a significant application. I guess I was wrong.

Things like Google Web Toolkit was around since 2006 , But on further read the interesting note is
"This application relates to U.S. Provisional Patent Application No. 60/714,285 filed on Sep. 6, 2005 entitled SYSTEM AND METHOD FOR SYNTHESIZING OBJECT-ORIENTED CODE INTO BROWSER-SIDE JAVASCRIPT.""

Does this bring a leagal question over all AJAX based platforms ? Anyway looks like Morfil Legal team is already in action

Well I was waiting for the days when the whole Rules Editor in Biztalk would be available over AJAX like front ends for business users. Ah looks like I will need to wait a little while longer..

BizTalk or WF vs BizTalk and WF

Well it is interesting to imagine what the future of a product will be that you love very passionaltely . I am of course talking about Biztalk . Ever since WF was announced and was shipped there was a question
Is WF going to replace Biztalk .
Is R2 going to be the last release of biztalk .
What is after R2 V next . What after VNext .
I have heard V Next is going to be based on WF .

That is a lot of questions and No I am not trying to answer them nor am I making any statements . This is just some random spaghetti thoughts that I feel I need to put out of my head .
Well for one I believe BizTalk as a product is NOT going away . It is very much here and it will be here to stay .I am no way discounting the argument that WF lets you do many of the things that Biztalk does today .But my point is how easy is it for you to do something that is already available in Biztalk . BizTalk today is much more than an orchestation engine . It has a very rich adapter story . Building a workflow engine is one thing having the surround story where you can integerate that engine easily with other systems is a totaaly different ballgame . Writing an EDI adapter for your project to connect using workflow project using hand built WF vs what you can achieve with BizTalk and the EDI Adpater that ships out of the box is two different set of efforts and they are very much different . Of course when you see the 23+ adapters shipping out of the box which includes SAP , MQ , Oracle, Peoplesoft , JD Edwards to name a few it is easy to see why I feel that way . Another importnat reason is today enterprises want to make thier integeration as smooth and integerated while being very cost effective . For this they need an integerated help desk . Case in point some enterprise EDI customers who had the Covast accelerator was not happy with the covast support that they recived compared to the level of support that they get for BizTalk . Considering EDI is not going anywhere with the advent of new standards like Rosettanet and eb-XML on the contarary EDI is growing in momentum and is growing about 6% YOY . So these customers will definiely buy a more integerated suite like Biztalk and not probably build or buy external party adapters .

The next release of Biztalk after R2 code named V Next is based on .net 3.0 stack where the WF that you build in Biztlak and the Visual studio will both be the same in fact the designer could be a single one . Also with adpters moving to the new WCF Framework . One could potentially see future adapters built on WCF being consumed by Biztalk ad well as WF . So will biztalk be replaced then , in future . Probably not again becuase Biztalk also does guaranteed delivery and has a high availability model buuilt in which is scalable vertically and horizontally which is again hard to replicate . But if you are a small shop and you want to send some SAP data to your big customer . You probably can use your SAP adapter based on WCF to send data while compromising on lot of enterprise class features at a much lower cost .

Now with R2's EDI support there would be massive RFID deployment at the edge and there should be some new RFID SKU like the previous partner edition whihc wil make the Biztalk footprint much larger . I know there are other RFID early movers out there but the reason I am willing to bet on BizTalk is I remember back in the 90's every game that I used to play from Wolfenstein to Jazz Jack Rabbit you need to install the sound driver for every game . same with a word processing you needed to install the driver for every printer which means the app maker has to figure out support for each printer /sound card vendor . But microsoft came out with a unified printer and sound driver model for which you could write your app and now you can plug and play any printer that you like . This is pretty much the model on RFID now so the RFID printer manufactures I am sure is gonna love that . The fact that RFID will be packeged with Biztalk is a master stroke that I gotta give it to Microsoft .

So in relaity both WF and BizTalk is going to remain playing to their own individual strengths .
WF in the smaller integeration efforts with lower cost for trimmed basic functionality and BizTalk giving a true integeration platform or the Service bus for the much larger enterprise.

BPEL for Windows Workflow Foundation March CTP

Microsoft has announced the new BPEL for Windows Workflow CTP .This is an add on the for WF .This provides import and export tools for BPEL and includes WF activities representing BPEL for the WF designer in VS 2005. Check it out here

MVP Summit 2007 Day -1

Had a real scare in the morning . I had my flight at 9:55 in the morning . at 8 in the morning I realize it is the new US DST switch today ( It used to be first sunday in April , now it is second sunday in March ) . Luckily Bini has all my bags packed . I got into the first pair of jeans and was rushing to the airport at a normal speed limit . Almost got pulled over by a cop :-) .
Flight was on time but since I did not have any big bags and corning being a smalll town was able to board the plane . There was some problem in my connection in Detroit. They made us change planes and got here in the evening . Cheked into the Renaissance , met a few Polish MVP's and then hit the Party with Palermo . The party was at jilliansbilliards a nice place to hang out . I met Jim Zimmerman, talked about his upcoming book on AJAX . Had a discussion on the value of hosting certain plugins available for visual studio into in an AJAX based browser container, we had some interesting discussionon WCF and VNext . Knocked around a game of pool and got chatting with Mateusz about who are the best programmers in the world . According to him it is Polish and he of course is from Poland :-) . He had a few arguments to his theory. He is presently planning a new game for kids .

Took a cab ,got back to the hotel and slept off early which is not anormal practice for me .But I think I will have a busy day tomorrow .

None of our BizTalk apps are facing any issue with DST change yet . I do not anticipate much changes with time put forward . It will be interesting to see how they behave when we fall back .

Encryption Decryption using BizTalk

Many have asked me how to configure biztalk with certificates and I tell them it is really easy . But many seems to have a big challenge with it. Sometimes it is the lack of proper understanding of the secrurity infrastructure . sometimes it is things like not setting up the thumbprint. I was planning on writing an article on the same . But today i saw a good post from Richard and I dont think I need to do that now .

This is a well documented excellent post

BIZTALK R2 and RFID

Biztalk R2 is nearing release and with support of RFID being delivered it will open a new area for biztalk apps altogether. I am excited about RFID as a technology though the technology is under fire including boycott by many activist group . Some of the concerns regarding privacy are valid but the potential of the technology is amazing. RFID or Radio frequency identification (RFID) is a data collection system based on tiny microchips attached to a box, pallet or individual item that communicate with other devices using radio waves. The RFID readers obtain data from the tags ( even write ) in bulk. Your RFID client software then collects , ortganizes and distributes this data . The tags can be active or passive meaning ones that are activated by sedning a signal or active which continously screams out its existence .
The benefits of RFID over existing technologies like Barcoding are many including more data capacity , do not need line of sight , bulk read , automated reading ,counting and life history of the objects using read/write tags
The applications are tremendous end to end visibility from factory to consumer , medicines, international shipments , animal tracking . From the Biztallk perspective we have an RFID Services running on the client platform with an RFID client on WCF (possibly ) shipping out of the bax whihc biztalk can leverage. Since this would be on WCF you should be able to consume this with not biztalk applications also .
The Event processing layer which hosts the Biztalk rules engine in some format will support things like eliminating duplicate reads , aggregation events or some custom threshhold rules.
I will be posting soon about Biztalk RFID soon . Meanwhile you can visit the RFID resource centre

BizTalk Server 2006 Large Maps

There is a little gem in 2006 that lets you control how large messages are transformed .

Previouslt in 2004 if you had a fairly large transformation , there i s nothing much you could do than scale up the server with more memory. In 2006 you could let BizTalk us the disk after a particular threshold , By default that value us set to 1 MB . That means if your XSLT transformation (which you realize is a tree transformation in memory, Your XML is loaded as the source tree and your declartive Xpath in the xslt is used to match the source tree matches to get the result tree) is larger than 1 MB BizTalk will use the large message transformation engine which will page to disk . This is nice feature for large maps but will slow you down ,Hey it is better than getting an OutofMemoryException and that Administrator's pager going off . You might want to tweak this value (disk is slower than memory ) , Of course you dont want it too high either to beat the whole purpose . You need to figure this value on your available capacity.

To do this you need to change the threshold in the registry
Goto 'HKLM\\Software\\Microsoft\\BizTalk Server\\3.0\\Administration'
and set the DWORD 'TransformThreshold'

Biztalk and EntSSO Issues

Time and again we run into Enterprise SSO issues with Biztalk databases.
If your database team itself is seperate it is little hard to figure out where exactly the problems lie . SSO issues are mostly realted to MSDTC settings .

It is not a common knowledge but there is an SSO configuration and troubleshooting tool . This does not fix any issue but just gives you a report . Get the tool here

Once you run the tool check the DTC settings .

MSDTC Settings:
Network DTC Access is enabled
Xa Transactions is enabled
Network Dtc Access Outbound is enabled
Network Dtc Access Inbound is enabled
Mutual Authentication required


Enterprise Single Sign-On Service:
Service Account: domain\mybtssvc
Account na\btssvc is a member of the following groups
BizTalk Application Users
BizTalk Isolated Host Users
BizTalk Server Administrators
Debugger Users
EDI Subsystem Users
IIS_WPG
SSO Administrators
SSO Affiliate Administrators
Administrators
Remote Desktop Users
Service status: Running

Hope that helped some of you .

Getting Custom Queries into HAT

You probably know that you can write your own custom HAT queries apart from the default HAT Queries that are available . I have seen people write their own little queries that they write or get from somewhere and use it to run some specific query often by copying into the text box of a show query window .

Did you know you can get your custom queries into HAT .See Here



If you want to do that it is not so hard. All the tracking queries are stored in
C:\Program Files\Microsoft BizTalk Server 2004\Tracking\Queries
or
C:\Program Files\Microsoft BizTalk Server 2006\Tracking\Queries as the case maybe or wherever you have your Biztalk installation directory .

You can make a copy of one of the templates and rename it to yourQueryName.trq and open the file in a notepad then you will see an
Modify the second value for the name you want to appear on the drop down.
and the fourth with your new Query .
don't change any of the other parameters .

Always make sure you test your query first in the HAT window before you do this .




Now the Pivot Table Part . This is the part where you are going to describe your result set and you start with the which you can leave as it is and
then come to the . You have to modify the PivotField each of which describes your result set each has certain elements

Name -- This is the Name of your Column
DataType !-- This is Data Type Date , Integer ,Number
PLDataOrientation !-- leave empty a.k.a No idea ,if U know leave comment :)
PLPosition !-- The column number like 1 ,2
EncodedType !-- Type adInteger,adVarChar
CompareOrderedMembersBy !-- UniqueName

You need to have < for < and > for > which is XML encoded text and namespace of x: , basically follow the template

Save and close and open your HAT . You should see your new query in HAT if everything went well . In case you get any error Close HAT and reopen the file and fix it . HAT actually locks the file . Hope that was of help

BPM and SOA conference - Day 4

Day 4

I opted to attend Jon's sessions as he always has this high energy when he speaks .
So true to that I picked two of jon's session
Developing Applications for the Web 2.0 and
Generating Orchestrations Using a Visual Studio Wizard.
The Orchestraion Wizard session was a complete dive in hands on on the tool . I was reallly blown away with the tool . This basically lets you templatize your solution , I tell you if you want to invest in 3 hours learning a feature this is one that you will absoultely not regret . Downlad the Wizad at Jon's Blog . The documentation is still in latest office . I send a feedback to Jon so hopefully they wil put an earlier office based documentation.



During Lunch I hanged out with Kevin(See Pic) whose team in corning I worked for pretty much the whole of last year going over the conference days so far . On day 4 there is only so much more presentaions you can take. I ended up going to " BizTalk WCF Adapters In-Depth " by Aaron Skonnard, Pluralsight and Gruia Pitigoi-Aron, Microsoft . This was a good session as it was not too many slide kind of session . I like those on Day 4 . The last one I picked was
"Part 6: The Future of the Microsoft Application Server Platform"
Steve Swartz & Clemens Vasters Microsoft . What I niceabout the session was it gves you a good understanding on what you can expect and what is going to come and the
road ahead like the transacted file systems , the transacted registry and even a transaction command from the command line " I must add Steve and Clemens have a great energy and they make a good team . I hae gone to many presentations where there are multple speakers but they never felt unfied . But these folks session are an absolute pleasure to attend and listen to ..


I spoke with Marjan our Biztalk MVP Community Lead on how the coference was a great value for everyone . I went to meet some of my friends and left seattle the next day morning and reached here withhout any major incidents en-route.



All in all it was a great conference and time well spend

BPM and SOA conference - Day 3

Day 3

I missed the first session, most part of it amyway Windows Workflow Foundation: Creating Custom Activities for Workflows , Remember the movie and the relaxing yesterday .Anyway I had downloaded all the labs that day which has a Custom Activty lab . i then took the Driving Business Process Automation through Vertical Accelerators Brennan O’Reilly & Mark Smith . The had some intersting case studies . Interesting I met Brennan through one of my old friends Vishnu . So that was a definite higlight .

The next session I picked was "Effective Techniques for Handling Large Messages in Service Oriented Solutions " Thomas Abraham. This was an intersting session and one thing that I picked up was how BTS2006 pages your maps for translation on to disk after a certain threshhold and how you can control that threshhold. .
The next session was BizTalk Server 2006 R2 Adapter Framework - Walkthrough
Chandramouli Venkatesh Microsoft , This was no kidding the most Packed session of the conference in my opinion . I was sitting on the floor as there were no seats available . So that kind of gives you a general idea. CV's session was a blast !!. he demoed a WCF based SAP adapter . This is one of my High value session in mt eval . Hey You now owe me that drink !

Well the best part about thursday was getting to go for Dinner with the Product Team . This had a pretty good crowd.

Oliver Sharp: General Manager – BizTalk Server
Steve Swartz: Architect – WCF
Patric McElroy: Product Unit Manager – BizTalk Messaging & Process Server
Dan Eshner: Product Unit Manager – BizTalk Executive & Analytics
Kris Horrocks: Sr Technical Product Manager – BizTalk Product Marketing
Peter Wise: Group Program Manager – BizTalk Customer Projects & Services
Kartik Paramasivam: Dev Lead - BizTalk Messaging & Process Server
Lee Graber: Dev Lead – BizTalk Messaging & Process Server
Andrew Layman: Product Unit Manager – BizTalk Interoperability
John Wyss: Group Manager - BizTalk Customer Projects & Services
Jean-Emile Elien: Architect - BizTalk Messaging & Process Server
Chandramouli Venkatesh - Adapters


THe conversation was spontaneous and I rememeber some of the jokes that I had on my cold . Well if you dont know Ravan is a an evil king in Hindu Mythology who has 10 heads. I was thinking what if Ravan catches cold !. It is pretty funny when you think about . And since Ravan has 10 heads one head of that being on the nect , it has 4 on one side and 5 on the other that creates an equilibirium problem to an otherwise symmetirical human structure . And like in SQL it is possible to have a split brain for an issue if you have 5 heads voting for and against. And another one I thought is Ravan does not need to do a seperate brain storning session . He is always in one . And finally if one head sneezes does the other heads go belss you !



I also got to meet most of the Biztalk MVP's whose blogs I read regularly like
Alan ,Bill , Jon,Matt , Rod , Scott , Stephen Thomas
.It was a great evening , it was in Daniel's Broiler in Bellevue Downtown and I tell you the lamb chops there are absoultely delcious not to mention the crab cakes . So was an exciting day in short and a memorable day in my MVP life.

BPM and SOA conference - Day 2

Day 2 :
The session started with Customer Panel: SOA Success Stories for me and I was realy amazed at the cariety and scale of challenges that Biztalk is solving . The next session thta i picked was
Avoiding 3 Common Pitfalls in Service Contract Design by Tim Ewald. This was a great session , This did focus on the facts that message contracts and schemas do evolve no matter how much reasearch you do upfront and the trick is to expect that change and how to make your schema extensible and how to change your idea of validation from a sub- set presence rather than an exclusion from a sub set of valid nodes.

There was a demo by K2 on thier new Workflow tool and I must say I am IMPRESSED with it's capabilities starting from AJAX based browser workflow designer to thier tight integeration into the Visual Studio environment and integerated debugging. It is code named BlackPearl , Reminds me of Pirates of the carribean . It is truly amazing howK2 has grown from it;s humble begininig in South Africa.

Choosing the next session was tough . I was torn between two sessions
Advanced Routing and Correlation with BizTalk Orchestrations
Lee Graber Microsoft and
Building an ESB on the Microsoft Platform
Brian Loesgen , Neudesic & Lukas Cudridge, Microsoft

I finally chose to attend Lee's Session and it was well worth it . His slide on Zombie factory ( classic Zombie a Recieve with a timer in the parallel branch ) had the room in splits :-) . Other thing that I really found out was Lee was not Chinese , i am not sure how that notion got into my head . This was one of my best rated sessions

I went for the "ESB Technical Deep Dive " by Brian next as I missed the intro talk but I was really impressed with the framework . You can get more details on the tool at EBIZTLKIT@microsoft.com .

I finished the day with a Developer Q and A and went to the Birds of the feather and Partner Expo. Some of the things that caught my fancey were

Farpoint Spread - An excel pipeline component

SKELTA workflow tool . They too has Ajax based workflow designer like K2 . It has outlook integeration too .It is not integerated into the .net platform at this point . This might be a cost effective option if K2 is out of your reach .

Java- Bizalk - JNBridge Adapter

Did stroll over to Jon's QuickLearn Booth , but couldnt catch him anywhere . Spend some time chatting at the BOF tables and went for a movie to relax

BPM and SOA conference - Day 1

Day 1: Went in Registerd for the conference and went to the keynote talk By David Chappel "Conference Keynote: SOA, BPM, and Microsoft: A Pragmatic View"
This was one of the best talk that I had at the conference . It really does give you a pragmatic view of things.

The next session was by John deVadoss on Real World SOA .This really had some intersting comparison on the big bang approach vs the small end to end scenario in finding out the SOA Value . This had some intersting videos on a leading bank in Australia and some good case studies.
There was a presentation over lunch by HP
After this one had to choose between differnt tracks and I kind of took
The Architecture of SOA - John Evdemon
Technical Solution Spaces for BizTalk and CSD - Oliver Sharp
Then we had an SOA Q and A Panel which was really valuble and it kind of had lo of thought provoking question like what is schema versionin if XML stands for Extensible Markup Languuage. Well that was not the exact question but you know the philosphical flamy question . Really felt lie after a days full of presentations. I had suddenly got this cold from somewhere by then . WEnt ove rto Bldg 3 to meet some of my old friends and watched some movie in the evening

BPM and SOA conference - Day 0

Day 0 : I had a kind of Bad flight from here to Seattle on US airways . After a really tough scrutinizing at security where they bought out everything from my aftershave , deo and my bottle of Dasani(Water) . I got on my flight from Elmira to Philly. I had a 45 minute connection which became a 20 minute connection as flight was late . So had a real tough time running from one terminal to another where they keep announcing my name wrong on the PA systems. Somehow made it to the terminal and was the last person in . The flight was long and I was hungry but could not buy food as they ran out of meals. ( Great first they charge you and then you font get it even if you want to buy ! ) . Reached Seattle and crashed at my friend DK's place after eating some nice Indian Pulav.

BizTalk on Virtual Server

There is considerable interest these days on virtualization and the benefits of virtualization in terms of bringing down the hardware and licensing cost .While I agree that the hardware cost and annual support cost does come down. The software licensing cost is not so straight forward. The answer really is it depends.

Let us say your organization A has 2 SQL Servers and 2 BizTalk servers. So that means you would have 4 Windows licenses to run those servers. I am assuming BizTalk and SQL are not on the same server.

Your present scenario is

BTS - 2
SQL - 2
WIN - 4


Now consider a virtualization scenario where you have a single Quad proc machine.
Now let us consider this has 4 virtual processors and 2 of them runs BizTalk, 2 of them runs SQL

Now in the old licensing model you need to pay license per processor that would mean you need to have

BTS - 4
SQL - 4
WIN - 4

Effectively by virtualization the license cost went up. But with the new windows licensing model you pay only if the virtual processor runs the Server, In our example that would mean

BTS - 2 ( as it is running only on 2 virtual processors )
SQL - 2
WIN - 4

So effectively your BizTalk and SQL licensing costs have not changed by virtualization but your actual hardware and annual support cost will be driven down considerably and you can take other advantages of virtualization. Let us say you want to emulate 6 virtual processors using your Quad proc machine and run 4 BTS and 2 SQL servers you can do that without buying any extra hardware by paying only the additional BizTalk server licensing cost . In this case

BTS- 4
SQL -2
WIN - 6

So as I said before virtualization does bring down support cost but not always software costs. But there are other software’s which are still using the physical processor based licensing. In that case you will have licensing benefits when you run more virtual processors than the actual processors and pay more when you have lesser virtual processors running your application.

BizTalk Server Goodies

There is a new BizTalk Server 2006 Troubleshooting Guide has been published. This has lot of troubleshooting techniques and tips. I found a few here . Higly recommended read . Get it here

There is also an Adapter comparison guide . Check out

Speaking at CNUG- DAM

I am speaking at CNUG- DAM . CNUG- DAM is a chapterr of chennai DotNet user group and is the only Design , Architecture and Management group in India

The topic for the talk is
"Common Enterprise Integration problems,
Design Patterns and Solutions using BizTalk "

Brief Agenda
Talk about some common problems faced in EAI world and how biztalk can be used to
address those effectively. Will be some scenario problems and design ,Splitter, Enricher , Aggregator , Normalizer,Different Convoy
Discuss one or scenarios with a demo ( Aggregation ) .
The problem of zombies
Q & A



The session is scheduled on 21 August - Monday between 7.00 p.m. to 9.00 p.m. in Sangeetha Residency.

Multiple Schemas in Custom Pipelines

I am loving Biztlak 2006 more and more . One of the cool feature of Biztalk 2006 which is proving a great help for me is the multiple pipeline in the same stage .

For eg let us you have a situation wher you have multiple flat files coming in and you need to find the schema . So normally we will put a flat file Dissasembler in and create a custom recive pipeline and we will need to create 3 pipelines.



In 2006 what you can do is have multiple flat file dissasembler components in the Dissassemble stage . So now your 3 pipelines now becomes one pipeline. All you need to for this is to go to your pipeline and drag in the 2 or 3 flatfile dissasembelrs and set the schema property to the ones you want and then voila. You wil notice that in the the Dissasemble stage property the execution mode appears as "First Match" .




But dont add too many dissasemblers in one stage . This wil have a negative imapct in performence . This is because probing for every dissasembler components degrades the perf.
In such a scenario consider implementing a Schema Resolver component

Visual Studio 2005 and Visual Source Safe

After spending quite a bit of time playing with Menus on my Visual Studio 2005 to find the option of "Add my solution to Source control " that was there in my prev VS version I realized that I had not installed Source control at all . This was weird as I did not remember explicitly unchecking any of the options except VJ# which i thought I was not going to need probably.

I ran setup again chose Add Remove and could not find the installtion Option.
Anyway after a few painful hours I have finally got it to work . This is what I had to do

1. ) Get the VS 2005 CD . Browse to the VSS folder and run Setup .
2. ) You need the first check box in the custom installtion . Everything else is really optional
3.) Install the Visual Source safe client and connect to your existing Sourcesafe Manully
4.) Fire up Studio, and go to Tools > Options > Source Control> Show all settings > Select the current source control plug-in to "Microsoft Visual SourceSafe"

Well I hope that helps some of you atleast ! If it did leave me a note !