So what makes internet content go viral? Are there a set of common factors that can ensure the success of your video or tweet? Is there a silver bullet? Over the last few years a number of companies, researchers and phony social media gurus have tried to explain the phenomenon. But it is still difficult to replicate the success of one campaign directly in another.

A paper by Berger & Milkman [pdf] looks at an interesting link between the emotion generated by the content and the degree of social transmission, or the virality factor. Articles which generated emotions like awe, anxiety or anger tended to be more viral than content which generated emotions such as sadness.

This brings me to why I am bothering with this right now. I have taken up an unpaid job with mindfulmum.co.uk after I finished my MSc in Computing Science. Being a startup I get the chance to work in multiple areas – from advertising to web design and optimization. But my main job is to deal with the social outreach of the site.

So I have started with the basics. Its only been a few days and we are looking to improve the site layout in such a way that it is easier to share content. Now we are looking into starting a campaign on facebook with the goal of increasing the audience of the site’s content. There is a lot of hard work still to be done and a site redesign is still in the works. I’m now thinking of ideas to create a good viral application which can launch the social aspect of the site.

Well, its only been a few days.

Quite a few of us have been using Google Instant for a while now. It is the natural progression of the ‘search suggestions’, after Google were convinced it could produce at least some useful predictions. Clearly this speeds up search and except a few irritating instances, it is a favourable development.

On the other side of things is the advertising and Search Engine Optimization.Every change that Google makes to its algorithms or UI is immediately dissected by all the specialists and professionals in these 2 fields. So what is their take on Google Instant?

I believe Instant has made SEO even more important. That is, of course, considering SEO in a positive light. Ensuring that a website actually do find themselves where they are supposed to be found on Google. Since Instant reduces the time people spend searching, it implies that being in the top-5 results is crucial. Earlier, being on the first page of Google’s results was good enough, while now it is looking likely that being in the first screen of Google is what is needed. The Click-through rates (CTR) for sites in the top 3 would be far greater than those below it. This increases the importance of good SEO of a website’s content.

How does this effect Google’s search advertising for each query? The Pay-per-Click(PPC) advertising works on a principle of first identifying an impression of the ad. An impression is noted whenever a user :

• Presses ‘Enter’
• Clicks on ‘Search’
• Selects a prediction
• Stays on page for >3 seconds
• Clicks on a result
• Clicks on a refinement (maps, news, latest

The 3 second window means that ads may get a 3 second window of display even though the query may not be the one which is advertised. Also, there will a be a number of key-words which will fall behind in the CTR counts due to the changes.

All that can be said just now is that this is not a game-changer, though it will upset a number of the previously accepted designs and trends in SEO and PPC.

As part of a Research Methods and Techniques course in Semester 1, we had to submit a literature review on a topic of our choice. I chose the above topic since I had an inclination towards how freshness of web search results was maintained. I got interested in the topic after reading the papers by Dong et al. on the topic. [3,8]. I got a decent grade but, obviously, there were a huge number of errors. I thought it would be useful to blog it – just to give me a reason to correct some of  the mistakes.

WARNING: This is a lot of text for one blog post.

Title: Identifying queries which demand recency sensitive results in web search

1. Introduction

Recency ranking in web search deals with the incorporation of temporal aspects of documents into the ranking model. The aim in recency ranking is to provide results to the user which reflect the freshness of the documents without compromising on the relevance of the results. Recency, as an aspect of web search, started gaining importance with the advent of news-search. More recently, it is the demand for real-time search results which has been driving research into ranking algorithms which take freshness of content into account.

Dong et al. [3] states that there are two major challenges in providing recency based ranking in web search:

• Firstly, The search engine has to identify which queries require recency sensitive ranking. This is important since recency sensitive ranking algorithms tend to perform poorly when used for queries where the retrieved data is relatively static.
• Secondly, once the query has been identified, ranking recently crawled content in the absence of rich historical data, such as click-through rates and in-links is a major challenge. This is because prestige based ranking algorithms like PageRank [7] give a lot of weighting to such features.

Both challenges are closely related since the method used to identify queries which need recency sensitive treatment must drive the method used in the ranking. In this literature review, we will be looking at the solutions proposed to solve the first challenge, though we will touch upon some relevant relations between the two stated problems. Broadly, the solutions are from the fields of news search [1, 4], time-series analysis [2, 3, 5] and data mining [6].

In Section 2, we will first examine the problem is detail. Section 3 deals with the critical examination of the methods used to solve the problem. And section 4, presents a conclusion which takes a holistic approach to the problem and tries to identify a research gap.

2. Problem Statement

In traditional information retrieval, documents are considered as static entities which do not change with time. This implies that the relevance of a result set associated with a query is also seen as a constant. But new content is generated on the web very frequently in the form of blogs, news, tweets and other user-generated content. Thus in web search, users demand results which reflect the recency of the content while maintaining the relevance of the result set. Thus recency ranking is indispensable.

One way of approaching recency ranking would be to introduce a feature, as part of the ranking algorithm, which gives a certain weight to the recency of a document irrespective of the query. But Dong et al. [3] have shown that recency algorithms perform poorly when applied to queries with a static information need. For example, “liver” is a static query. The result set remains fairly constant year-on-year. Applying recency ranking to this query will result in a long established result losing out. Hence there is a need for identification of recency sensitive queries before ranking.

When a user enters a query they may or may not specify whether their intent is recency sensitive. For example, the intent behind a query such as “earthquake” could be interpreted in two ways. The user may be looking for geological information regarding earthquakes in general or they may be looking for information regarding a specific, recent earthquake. Since users do not clearly specify their information need in terms of recency, identification of such queries becomes a challenging problem.

It is not possible to identify the intent of the user query by analysing the query on its own. Thus, the methods used to tackle this problem consider more other sources to perform recency classification of the query.

3. Methods Used to Solve Problem

The methods proposed to solve the above problem can be broadly divided into three classes. They are:

1. News Vertical solutions : This consists of methods which are focused only on the news vertical. The solutions proposed in [1, 4] perform well on news-related queries but have not been proved successful on a more general query.
2. Temporal Modelling : It is often difficult to capture the recency intent of a user using only a single query. Methods put forward in [2, 3, 5 ] deal with either modelling of queries and time spaces looking for discrepancies from the normal frequencies. Discrepancies can be viewed as reason to provide recency sensitive results.
3. Composite model : Zhang et al. [6] proposes a model which takes into account the time series, query log analysis, click through data and other features and feeds it into a machine learning model to produce a classifier. Using all the above measures, the classifier can identify queries which require recency sensitive ranking.

3.1 News Specific Solutions:

There are 2 closely related approaches in this class. Diaz [1] and Konig et al. [4] both aim to find the same solution to the problem. Both aim to predict whether a user will click on the dedicated news, if displayed, for a particular query. The approaches taken by both are different and they have their advantages and disadvantages, Diaz [1] proposes a method which uses the fact that there exists a relation between click-through rates and recency-sensitive, news related queries. Here first the probability of a user clicking the separate news display if it is indeed displayed for a particular query, is estimated. An initial guess at the probability is made from the context of the query is made. This is then presented as a Beta distribution such that it depends on both the assumption as well as past clicks.

Query similarity is considered and then since the probability is defined, a threshold value is set. If the probability of the query getting clicks is higher than the threshold, then results are shown. This threshold value is leniently set so that more queries have an opportunity to get exposure. Most importantly, different from the approach taken by Konig et al. [4], click feedback is used to correct any errors made by the algorithm. It is shown experimentally that in case of an erroneous judgement regarding the showing of news results, since that result would not receive any clicks, the judgement will be reversed in time automatically.

It was seen that using click feedback improves on the baseline taken in the experiments. The leniency introduced in the threshold value did not improve the overall performance. This was due to poor classification of queries which were under the threshold value.

In the method proposed by Konig et al. [4], first a set of features are considered from the corpora. Then the corpus is defined to include wikipedia, blogs and news. After feature extraction, like tf-idf, cosine similarity, Jensen-Shannon divergence, the learning model is applied. The learning model used is the Multiple Additive Regression Trees model or MART. It is built on Stochastic Gradient Boosting. It is claimed that MART has better accuracy and can handle the rich feature set provided it. Randomness is added to the algorithm to improve robustness. This algorithm is then applied to the training data and after making sure that there is no overlap between the training data and the test-collection, the testing is done. Quantitative CTR prediction for future queries is observed to be very accurate. PR curves for the threshold CTR values shown below (as seen in paper). Overall prediction success = 82%.

The major difference between the methods proposed by Konig et al. [4] and Diaz [1] lie in the usage of temporal features and machine-learning algorithms used. Both methods are biased towards news related queries and so fail to answer questions relating to coverage of different queries.

3.2 Temporal Modelling

This class of methods approaches the problem by modelling the temporal aspects of either the queries or the results of the queries to identify which ones are recency-sensitive.

The methods put forward by Vlachos et al. [5] and Dong et al. [3] are opposite in their approach. Vlachos et al. [5] proposed a method that follows the topic frequency over different time-slots, detecting any discrepancies in the frequency or “bursts” of activity. But in the methods proposed by Dong et al. [3], time slots are modelled and compared with each other.

Vlachos et al. [5] introduce methods to identify important features in the time series and also to identify bursts in activity. The paper aims to match queries with temporal bursts in activity. The aim of their compression algorithm is to minimize the Euclidean distance between the query and the time-series representation. Best Fourier coefficients are used to represent the compression because they describe the series better. 3 algorithms are put forward to improve the similarity search of the query with the compressed time-series – BestMin, BestError and BestMinError.

A cumulative distribution function of the time-series is used and the high-frequency of queries during important periods will be significantly off from the mean. Thus the periods with maximum query activity are identified. Finally, to identify bursts in query activity which are out of the ordinary, first identify and then compress the data to enable comparison later. The storage can be done on any relational database. For the detection phase, moving averages are used over the time-series. If a power value was far higher that the moving average, then it was deemed to be a burst. Next, to find all queries with similar burst patterns, the features and compressed and then compared. Similarity and overlap between the bursts are checked. Thus it becomes possible to identify a set of queries which show a spike in activity.

Dong et al. [3] proposed a method which aimed to classify a classify a query as recency sensitive. Time-slots, as opposed to topics or queries, were modelled. N-gram models were made for content and queries in each time slot and compared with past time-slot models. Any discrepancy in content and query models would then show as a recency trend. A final ‘buzz’ score is computed for every query. If this value exceeds a certain threshold, the query is considered recency sensitive. To determine the ideal constants of the function, some manual classification was done. Through the reported experiments we see that the query classifier has an accuracy approaching 90% at times when there is high breaking-news traffic. At other times the classifier doesn’t work as well.

Finally, as part of the recency ranking framework, four separate recency features were considered and compared -Timestamp, Linktime, WebBuzz and Page classification. The ranking used machine learning methods using recency training data as well as regular relevance training data. GBrank was the machine-learning algorithm used. Three ranking models were also tested – Composition Model, Overweighting Model and the Adaptation Model.

Another method which defines a query temporal model is proposed by Diaz & Jones [2]. This paper looks at the temporal nature of query results with the aim of predicting the relevance of the result set. A temporal profile is made for the query based on timestamps of the top 5 documents retrieved. The granularity is defined at 1 day. Smoothing of the model is required since some days may have an awkward spike in query traffic while other days may have no traffic at all. This concept of smoothing has been discussed in time series analysis. Once the model is prepared, features are extracted to predict the relevance of the result set. This method of modelling queries is both simple and elegant from a theoretical point of view. From the context of this paper, it was effective as well.

3.3 Composite Modelling

While the above models considered certain models and concepts to solve the recency-sensitive query identification problem, none have combined all the relevant metrics to see which ones perform the best.

In the paper by Zhang et al. [6], a very good coverage of metrics is seen to solve a unique problem. A number of queries occur at regular time intervals but require recency sensitive handling. E.g. “SIGIR”. In 2010 – the results should focus on SIGIR2010 and not on past conferences. These queries are called Recurrent Event Queries, or REQ. The time at which these queries occur is predictable and they account for 6% of all queries issued and hence are worth dealing with.

The paper presents an REQ classifier which uses a wide range of features together. Also they are all combined by machine learning algorithms. The paper defines 3 different machine learning algorithms and compares them to find the most effective.

The paper uses a number of features which it then applies to a machine learning framework. First deals classifying queries into ones with explicit time, implicit time, and no time markings. Query log analysis is used to obtain metrics like whether a query has an implicit or explicit temporal nature, the frequency of that query, how many different years are has that query been on for and a chi-square distance. Metrics on changes to queries done by users during a session are also included. Click log analysis is used to get the click-through rates of the queries. The result sets of implicit time queries are checked as to whether they show any year features. A list is made with a series of REQ words which are tokenized from implicit queries.

Three different learning algorithms are used to combine the above metrics – Naive Bayes, SVM [9] and Gradient Boosted Decision Tree (GBDT). For the purpose of evaluating the effectiveness of the learning algorithms, 6000 queries were marked as REQ or not manually by human editors.

After running the algorithms evaluation was done using a PR graph. Precision was determined as REQ/(total classified REQ). Recall was defined as REQ/(all REQ). Finally from the experiments it was found that the Gradient Boosted Decision Tree machine learning algorithm was the one which performed the best. The ratio of the no. of times a query is issued with year a qualifier to without, was the most important feature. And finally 3.6% gain in DCG at the top position was observed against regular existing web search.

4. Future Scope and Research Gap

There is work still to be done in the field of recency query identification. Twitter is a source of real-time news. It still needs to be investigated if mined Twitter data can be used effectively to identify recency sensitive queries. Also, incorporating data from book-marking sites like Digg or Del.io.us can be used as an indicator of recency sensitivity of a related query.

5. Bibliography

[1] Fernando Diaz. 2009. Integration of news content into web results. In Proceedings of the Second ACM International Conference on Web Search and Data Mining (WSDM ’09), Ricardo Baeza-Yates, Paolo Boldi, Berthier Ribeiro-Neto, and B. Barla Cambazoglu (Eds.). ACM, 182-191. http://doi.acm.org/10.1145/1498759.1498825

[2] Fernando Diaz and Rosie Jones. 2004. Using temporal profiles of queries for precision prediction. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval (SIGIR ’04). ACM, USA, 18-24. http://doi.acm.org/10.1145/1008992.1008998

[3] Anlei Dong, Yi Chang, Zhaohui Zheng, Gilad Mishne, Jing Bai, Ruiqiang Zhang, Karolina Buchner, Ciya Liao, and Fernando Diaz. 2010. Towards recency ranking in web search. In Proceedings of the third ACM international conference on Web search and data mining (WSDM ’10). ACM, USA, 11-20. http://doi.acm.org/10.1145/1718487.1718490

[4] Arnd Christian Konig, Michael Gamon, and Qiang Wu. 2009. Click-through prediction for news queries. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval (SIGIR ’09). ACM, USA, 347-354. http://doi.acm.org/10.1145/1571941.1572002

[5] Michail Vlachos, Christopher Meek, Zografoula Vagena, and Dimitrios Gunopulos. 2004. Identifying similarities, periodicities and bursts for online search queries. In Proceedings of the 2004 ACM SIGMOD international conference on Management of data (SIGMOD ’04). ACM, USA, 131-142. http://doi.acm.org/10.1145/1007568.1007586

[6] Ruiqiang Zhang, Yuki Konda, Anlei Dong, Pranam Kolari, Yi Chang, and Zhaohui Zheng. 2010. Learning recurrent event queries for web search. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP ’10). Association for Computational Linguistics, Morristown, NJ, USA, 1129-1139. http://portal.acm.org/citation.cfm?id=1870658.1870768

[7] PageRank

[8] Twitter for recency ranking dong et al.

[9] Corinna Cortes and V. Vapnik, “Support-Vector Networks”, Machine Learning, 20, 1995. http://www.springerlink.com/content/k238jx04hm87j80g/

Hi Gabor,

I am the original author of GNU grep.  I am also a FreeBSD user,
although I live on -stable (and older) and rarely pay attention
to -current.

However, while searching the -current mailing list for an unrelated
reason, I stumbled across some flamage regarding BSD grep vs GNU grep
performance.  You may have noticed that discussion too...

Anyway, just FYI, here's a quick summary of where GNU grep gets
its speed.  Hopefully you can carry these ideas over to BSD grep.

#1 trick: GNU grep is fast because it AVOIDS LOOKING AT
EVERY INPUT BYTE.

#2 trick: GNU grep is fast because it EXECUTES VERY FEW
INSTRUCTIONS FOR EACH BYTE that it *does* look at.

GNU grep uses the well-known Boyer-Moore algorithm, which looks
first for the final letter of the target string, and uses a lookup
table to tell it how far ahead it can skip in the input whenever
it finds a non-matching character.

GNU grep also unrolls the inner loop of Boyer-Moore, and sets up
the Boyer-Moore delta table entries in such a way that it doesn't
need to do the loop exit test at every unrolled step.  The result
of this is that, in the limit, GNU grep averages fewer than 3 x86
instructions executed for each input byte it actually looks at
(and it skips many bytes entirely).

See "Fast String Searching", by Andrew Hume and Daniel Sunday,
in the November 1991 issue of Software Practice & Experience, for
a good discussion of Boyer-Moore implementation tricks.  It's
available as a free PDF online.

Once you have fast search, you'll find you also need fast input.

GNU grep uses raw Unix input system calls and avoids copying data
after reading it.

Moreover, GNU grep AVOIDS BREAKING THE INPUT INTO LINES.  Looking
for newlines would slow grep down by a factor of several times,
because to find the newlines it would have to look at every byte!

So instead of using line-oriented input, GNU grep reads raw data into
a large buffer, searches the buffer using Boyer-Moore, and only when
it finds a match does it go and look for the bounding newlines.
(Certain command line options like -n disable this optimization.)

Finally, when I was last the maintainer of GNU grep (15+ years ago...),
GNU grep also tried very hard to set things up so that the *kernel*
could ALSO avoid handling every byte of the input, by using mmap()
instead of read() for file input.  At the time, using read() caused
most Unix versions to do extra copying.  Since GNU grep passed out
of my hands, it appears that use of mmap became non-default, but you
can still get it via --mmap.  And at least in cases where the data
is already file system buffer caches, mmap is still faster:

  $ time sh -c 'find . -type f -print | xargs grep -l 123456789abcdef'
  real	0m1.530s
  user	0m0.230s
  sys	0m1.357s
  $ time sh -c 'find . -type f -print | xargs grep --mmap -l 123456789abcdef'
  real	0m1.201s
  user	0m0.330s
  sys	0m0.929s

[workload was a 648 megabyte MH mail folder containing 41000 messages]
So even nowadays, using --mmap can be worth a >20% speedup.

Summary:

- Use Boyer-Moore (and unroll its inner loop a few times).

- Roll your own unbuffered input using raw system calls.  Avoid copying
  the input bytes before searching them.  (Do, however, use buffered
  *output*.  The normal grep scenario is that the amount of output is
  small compared to the amount of input, so the overhead of output
  buffer copying is small, while savings due to avoiding many small
  unbuffered writes can be large.)

- Don't look for newlines in the input until after you've found a match.

- Try to set things up (page-aligned buffers, page-sized read chunks,
  optionally use mmap) so the kernel can ALSO avoid copying the bytes.

The key to making programs fast is to make them do practically nothing.  

Regards,

	Mike

Check the sub-links within the first result. What is that link below Login doing there?!

Ryan Aguillard.
It links to the  facebook internationalization page( http://www.facebook.com/index.php/Internationalization ) which redirects to my home page.

What is that doing there, Google?

Who is Ryan Aguillard?

A quick Google & facebook search yields no results. All the links are generic. The only links which seem promising are a forum post – with no replies by the way – asking the same question. Also, a link to a Huffington Post profile which looks unused.

Update 1:

Recently (Aug 20, 2010), Google announced that they will be showing increased number of results from each domain, if the search is focused enough on a single domain.

They said, “Today we’ve launched a change to our ranking algorithm that will make it much easier for users to find a large number of results from a single site. For queries that indicate a strong user interest in a particular domain, we’ll now show more results from the relevant site. We’re always reassessing our ranking and user interface, making hundreds of changes each year. We expect today’s improvement will help users find deeper results from a single site, while still providing diversity on the results page.”

The aberration that is Ryan Aguillard is probably a result of this new algorithm doing something stupid.

Update 2:

I received a huge spike in traffic yesterday on Aug 31st 2010.
It also seems the problem has been set straight on Sept 1st 2010.
I guess thats the end of that.

The research paper, titled Usable Gestures for Mobile Interfaces: Evaluating Social Acceptability by Julie Rico and Stephen Brewster, looks at what gestures people will be willing to make using mobile devices in public places.

The on-the-street user study demonstrated that user acceptance of gestures is increased after even one positive experience. The survey demonstrated the important role that observers play in social acceptability, with highly acceptable gestures including subtle imitations of everyday gestures and gestures with highly visible cues demonstrating their role as an interaction with an interface. These results provide researchers with concrete tools that can be used to assess the social acceptability of multimodal interaction techniques at an early stage of development.

Case in point now is the much hyped Sixth Sense device designed by the Indian, Pranav Mistry. In one of the examples in the demo, he shows the device projecting something onto another person. As Nick Jones points out, how many people would enjoy that? How long would you last before you get punched by the boyfriend of the girl whose T-Shirt you’re reading from?

Coolness of the product should not blind us from the real utility of the product. It should also not blond us from the acceptability of what we do with it.

I hope mobile device designers read this and not make people openly point at people or things, or make us do idiotic gestures.

If anyone wants to read the chat-logs you can find it here.

So, here goes!

The first page which you can use is the Ubuntu wiki page – UsingDevelopmentReleases. It has tonnes of info and links that you can use during your development time. The first thing you’ll need is an environment to develop in. A place where whatever crap you end up doing isn’t going to damage your happy Ubuntu installation. One of the things you can use is chroot

Installing chroot

Easy thing, this. Run these commands to get the installation done
sudo apt-get install chroot
sudo apt-get install debootstrap

If you haven’t already enabled “Source Code” and “Universe” in System → Software Sources → Ubuntu Software you can do that now.

Note: We’ll be dealing with some basic bug-fixes etc first. This is not going to be gnome centric (which is a natural assumption that many will have considering its Ubuntu)

Run the following command :

sudo apt-get install --no-install-recommends bzr-builddeb ubuntu-dev-tools fakeroot build-essential gnupg pbuilder debhelper

This will give you a bunch of tools that are going to be useful, generally (not just in these examples)

bzr-builddeb pulls in bzr, which is the short for Bazaar which we’ll use to get the source-codes for one or two examples.
ubuntu-dev-tools pulls in devscripts which are incredibly helpful at making repetitive packaging tasks easy
fakeroot is needed by debuild (in devscripts) to mimic root privileges when installing files into a package
build-essential pulls in lots of useful very basic build tools like gcc, make, etc
gnupg is used to sign files in our case (uploads in the future)
pbuilder is a build tool that builds source in a sane, clean and minimal environment it sets up itself
debhelper contains scripts that automate lots of the build process in a package

Let’s first set up our gpg key which we use to sign files for upload. More generally it is used to sign and encrypt mails, files or general text. We use it to indicate that WE were the last to touch a file and not somebody else and that ensures that only people who we know about get to upload packages.

If you have no gpg-key yet, run the following command :

gpg --gen-key

When you are doing this, it is completely fine if you stick to the defaults. You don’t have to comment, for example. Enter your name, email address and just stick to the default values for now. It could be that gpg is still sitting there and waiting for more random data to generate your key – that’s expected and fine. Just open another terminal while we carry on, it’ll finish on its own. As said earlier: if you have a gpg key already, skip this step

If you need more information on gpg-keys you can look through the Gnu Privacy Guide which about everything in more detail.

Now we can set-up pbuilder.

Open an editor and edit the file ~/.pbuilderrc (create it if you don’t have it yet) and add the following content to the file:

COMPONENTS=”main universe multiverse restricted”

Save it and once you’re done, run the following command :

sudo pbuilder create

What does pbuilder do?

It builds packages in a clean and minimal environment.
It keeps your system “clean” (so you don’t install millions of build dependencies on your own system).
It makes sure the package builds in a minimal, unmodified environment so you ensure that the package does not just build because you made lots of changes on your system, but the build is reproducable

You can update package lists (later on) with:

sudo pbuilder update

To build packages you run:

sudo pbuilder build package_version.dsc

How does pbuilder work? It first gets the minimal packages for a base system and stores them in a tarball. Whenever you build a package it’ll untar the base tarball, then install whatever your current build requires, build it, then tear it all down again. Luckily it caches the packages.

Something else we can do in the meantime is if you use the bash shell, which is the default, please edit ~/.bashrc. At the end of it, add something like
DEBFULLNAME=”Your Name”
DEBEMAIL=”your.mail.id@provider.com”
Once you’re done editing ~/.bashrc, please run source ~/. bashrc (it’s only needed once)

QUESTION: Should these match the values we put into GPG?
A: Yes

Ok, with this out of the way, the packaging tools will know you by your name and you don’t need to enter it, for example if you do changelog entries, etc.

(aside) General Fundas:

Ubuntu is very special in how it’s produced and how we all work. As you know it comes out every 6 months and that means we have a tight release schedule and everything we do and work on is defined by that schedule. Check this out for the current release schedule for maverick.

In that link, basically, green means: lots is allowed here, red means: almost nothing is allowed here.

In more detail,

- toolchain is uploaded for the new release (gcc, binutils, libc, etc.), so the most basic build tools are there
- new changes that happened in the meantime are synced or merged (more on that later on)
- ubuntu developer summit (uds) happens where features are defined and talked about
- up until debian import freeze we import source changes from debian semi-automatically
- up until feature freeze we get new stuff in, work on features, try to make it all work
- if a feature is not half-way there yet by feature freeze, it will likely get deferred to the next release
- from feature freeze on you can see that lots of freezes are added throughout the weeks and you’ll need more and more freeze exceptions for big changes

The focus is clearly: testing, testing, testing and fixing, fixing, fixing

So How do you get stuff in?
Only people who “we know” get to upload packages directly, as we said before. This means that as a new contributor you will have to work with sponsors who basically review your work and upload it for you. Once you did that a couple of times and they recognise you and your good work, you can apply for ubuntu developer membership and you can ask the people you’ve worked with for comments on your application

It’s not very complicated, you basically set up a wiki page with your contributions, ask for comments and submit for an #IRC meeting of the developer membership board and you’re done. No need to learn a secret handshake, send me money or anything else. It’s contributions and good work that counts.

Now get the source of the hello package and run the command:

apt-get source hello
sudo pbuilder build hello_*.dsc

Now, pbuilder will work as explained earlier.
Note : .dsc has some metadata like checksums and the like. Not important right now.

You can check out the contents of /var/cache/pbuilder/result and it will contain the built hello.deb file.

End of Part 1 – There is no logical end here, but I don’t like very (extra) long posts.

Some time back I asked a question on Quora : How does a computer choose a Random Number?

A number of interesting responses were there, which I’d like to share here.

The Question

Since a computer can’t choose a number truly at random (and neither can humans, technically), what is a random number in the context of a computer. Since the computer calculates the number, is it only “relatively random” i.e. to the user, the number ‘seems’ random when in fact it isn’t? I was wondering about whether a user can successfully predict which “random” number is going to be generated by the computer.

The Answers

I got a few ansers, the best and most explanatory of which was by Kiat Chuan Tan.

There are two main ways a computer can choose a “random” number: using a pseudo-random number generator (PRNG), or using a hardware random number generator.

A pseudo-random number generator, as the name suggests, isn’t truly random. PRNGs typically use deterministic algorithms such as lagged Fibonacci generators or the Mersenne twister. These numbers generated by these algorithms are completely determined by the start state, and will eventually repeat. However, a good choice of parameters for the algorithms will make the period sufficiently large for practical use (2^19937 – 1 for the Mersenne twister).

A hardware random number generator, on the other hand, is theoretically supposed to be truly random, in the sense that it is based on noise generated by physical processes, e.g. sampling of ambient noise from a sound card.

Micheal Hamburg added that:

There are some generators that (unlike the Mersenne Twister) are believed to be cryptographically pseudorandom.  That is, if they start with a small amount of randomness (say, 32 bytes), then there should be no algorithm which, running on all the supercomputers in the world for a thousand years, could notice a meaningful difference between the “random” numbers they output and truly random numbers generated by a hardware device.

Now, I was wondering if I could actually find a service which could, essentially, guarantee me randomness. A source for random numbers which was truly random and unpredictable.

Random.org

Random.org is an interesting service operated by Mads Haahr of <!– who is a Lecturer in –> the School of Computer Science and Statistics at Trinity College, Dublin in Ireland.

So how do they generate their random numbers?

RANDOM.ORG uses radio receivers to pick up atmospheric noise, which is then used to generate random numbers. The radios are tuned between stations. A possible attack on the generator is therefore to broadcast on the frequencies that the RANDOM.ORG radios use in order to affect the generator. However, radio frequency attacks of this type would be difficult for a variety of reasons. First, the frequencies that the radios use are not published, so an attacker would have to broadcast across all frequencies of all bands used for FMAM broadcasting. Second, this is not an attack that can be launched from anywhere in the world, only reasonably close to the generator. RANDOM.ORG currently has radio receivers in several different countries, which would make it difficult to coordinate this type of attack. Third, if an attacker actually did succeed at broadcasting highly regular signals (e.g., perfect sine waves) at exactly the right frequencies from the right locations, then the RANDOM.ORG real-time statistics would pick up the drop in quality very rapidly. In particular, the Source Purity and Information Entropy tests would start failing dramatically, which would raise an alert.

The need for such a tool is greater in Linux than in Windows considering you have a greater degree of freedom in configuring obscure files to make the OS perform the way you want it to. From that perspective, it has become quite a pain searching for files of something I had installed a long time ago. Also, I’ve been reading a few research papers of late and I’ve been making some notes in txt. ReFinding something is a nightmare if you can’t remember the filename.

So I started looking into the different desktop search options I had.

Beagle

Beagle is widely considered to be the leading tool for desktop search on Ubuntu. Its written in C# and runs using Mono. It uses Lucene’s C# implementation for indexing and it seems to be good. It was launched with (a bit of) fan-fare back at the 2004 Guadec Conference in Norway. It is quite similar to Apple Searchlight and was, interestingly, announced on the same day as Searchlight. Norway being 6+ hours ahead of the United States, it was technically released earlier!

But Beagle faced performance issues regarding its ‘crawling’ process which was very processor heavy. Users grew irritated with the constant processing involved in keeping a fresh index that many disabled the tool. Some time ago development on the Beagle Project was stopped and so, I believe, did the adoption by new users.

As a product, though, Beagle is quite good. It has a rich feature set and if you ignore the index refresh issues, its quite good at retrieval. It indexes all imaginable file types and has some interesting features, like the support of glob expressions which I haven’t found in other search engines (except Recoll).

Tracker

Tracker is short for Meta Tracker. It is a light and fast desktop search implemented using C. Tracker’s lightness has meant that it doesn’t index the innards of some types of files – pdf, ppt, mailboxes etc. But it is faster and doesn’t eat up processor cycles, so its a more quick and dirty solution. And of course if you can’t find something using it, you can always switch over to good ol’ grep.

Earlier, documentation was very poor on Tracker, but it has improved, I feel.

Google Desktop Search

Google Desktop Search isn’t open source, so it doesn’t get brownie points. But it is quite useful.  It indexes almost all file-types including pdf and ppt. The engineering behind it is good and it doesn’t hog (too much) processor time. If you can get over the fact that Google will get even more of your information, then this is the desktop search tool for you. I prefer to have at least a bit of my own life to myself, so I’m left with very few options.

There are many other tools which I haven’t mentioned here.

Terrier is one which I haven’t tested out in detail yet. I will be looking through the code of this, though, for obvious reasons. It isn’t built with desktop search in mind, but might perform well.

Recoll is one which I haven’t tried out fully yet. It runs with a Xapian back-end of which I know very little of.